Sparsifier

Make your neural network sparse

Overview

A sparse vector, as opposed to a dense one, is a vector which contains a lot of zeroes. When we speak about making a neural network sparse, we thus mean that the network’s weights are mostly zeroes.

With fasterai, you can do that thanks to the Sparsifier class.

source

Sparsifier

def Sparsifier(
    model:nn.Module, granularity:str, # Granularity of sparsification (e.g., 'weight', 'filter')
    context:str, # 'global' or 'local'
    criteria:Criteria, # Criteria to determine which weights to keep
    nm:bool=False, # Use an N:M sparsity pattern (forces 2:4)
    layer_type:Type[nn.Module]=Conv2d,
    data:NoneType=None, # Calibration data for activation-based criteria (e.g., wanda)
):

Class providing sparsifying capabilities

The Sparsifier class allows us to remove some weights, that are considered to be less useful than others. This can be done by first creating an instance of the class, specifying:

  • The granularity, i.e. the part of filters that you want to remove. Typically, we usually remove weights, vectors, kernels or even complete filters.
  • The context, i.e. if you want to consider each layer independently (local), or compare the parameters to remove across the whole network (global).
  • The criteria, i.e. the way to assess the usefulness of a parameter. Common methods compare parameters using their magnitude, the lowest magnitude ones considered to be less useful.

Key Methods

User can pass a single layer to sparsify by using the Sparsifier.sparsify_layer method.

source

Sparsifier.sparsify_layer

def sparsify_layer(
    m:nn.Module, sparsity:float, # Target sparsity, a fraction in [0, 1] (0.4 = 40%)
    round_to:int | None=None, # Round to a multiple of this value
)->None:

Apply sparsification to a single layer

Most of the time, we may want to sparsify the whole model at once, using the Sparsifier.sparsify_model method, indicating the fraction of weights you want to remove, e.g. 0.5 for 50%.

source

Sparsifier.sparsify_model

def sparsify_model(
    sparsity:float | dict, # Target sparsity, a fraction in [0, 1] (0.4 = 40%), or a per-layer dict
    round_to:int | None=None, # Round to a multiple of this value
)->None:

Apply sparsification to all matching layers in the model

Advanced Options

In some case, you may want to impose the remaining amount of parameters to be a multiple of a given number (e.g. 8), this can be done by passing the round_to parameter.

Instead of passing a single value of sparsity, a dictionary of per-layer sparsities can be provided. This allows fine-grained control over which layers get sparsified and by how much.

Example: Apply different sparsity levels to specific layers:

sparsity_levels = {
    'conv1': 0.3,           # 30% sparsity on first conv
    'layer1.0.conv1': 0.5,  # 50% sparsity
    'layer2.0.conv1': 0.7,  # 70% sparsity (more aggressive)
}
sparsifier.sparsify_model(sparsity=sparsity_levels)

Every ratio is a fraction in [0, 1] — a percentage such as 50 is read as 0.5 for one release, with a FutureWarning.

See Also

  • SparsifyCallback - Apply sparsification during fastai training
  • Criteria - Different importance measures for selecting what to sparsify
  • Granularity - Control what gets sparsified (weights, filters, etc.)
  • Schedules - Control sparsification progression during training
  • Pruner - Structured pruning that removes filters entirely
  • Parametrize - The master weight this class masks, snapshots and rewinds when something else computes the weight

Tests live in nbs/tests/test_sparsifier.ipynb.