Regularize Callback

Perform Group Regularization in fastai Callback system

Overview

The RegularizeCallback applies structured regularization during training to encourage weight sparsity at various granularities. This is useful as a pre-pruning step: by regularizing groups of weights toward zero during training, subsequent pruning can remove more parameters with less accuracy loss.

Key Features: - Supports every granularity defined in fasterai.core.granularity.Granularities (for Conv2d: 'weight', 'shared_weight', 'channel', 'column', 'row', 'kernel', 'filter', and their shared_/slice variants) - Compatible with any criteria from fasterai.core.criteria - Optional scheduling to vary regularization strength over training

---

source

RegularizeCallback

def RegularizeCallback(
    criteria:Criteria | list[Criteria], # Importance criteria, e.g. large_final
    granularity:str | list[str], weight:float=0.01,
    layer_types:Type | list[Type]=Conv2d, # Module types to regularize
    schedule:Schedule | None=None, # Optional schedule for the weight
    verbose:bool=False, # Report the weight after each epoch
):

Basic class handling tweaks of the training loop by changing a Learner in various events

Parameters: - granularity: Level at which to group weights, from Granularities (e.g. 'weight', 'channel', 'kernel', 'filter', 'layer') - weight: Regularization coefficient (higher = stronger regularization)

---

Usage Example

Apply filter-level L1 regularization to encourage entire filters to become unimportant (making them easier to prune later):

from fasterai.regularize.regularize_callback import RegularizeCallback
from fasterai.core.criteria import large_final

# Apply L1 regularization at filter granularity
cb = RegularizeCallback(
    criteria=large_final,
    granularity='filter',
    weight=0.01,
    verbose=True
)

learn.fit(10, cbs=[cb])

Typical Workflow: 1. Train with RegularizeCallback to push unimportant filter groups toward zero 2. After training, use PruneCallback or Pruner to remove the zeroed-out structures 3. Fine-tune the pruned model to recover any lost accuracy

---

See Also

• Sparsifier - Apply sparsification after regularization pushes weights to zero
• Criteria - Importance measures that can leverage regularized weights
• SparsifyCallback - Combine with sparsification for gradual pruning

Tests live in nbs/tests/test_regularize_callback.ipynb.