Sparsifier

Make your neural network sparse

Overview

A sparse vector, as opposed to a dense one, is a vector which contains a lot of zeroes. When we speak about making a neural network sparse, we thus mean that the network’s weights are mostly zeroes.

With fasterai, you can do that thanks to the Sparsifier class.

source

Sparsifier


def Sparsifier(
    model:nn.Module, # The model to sparsify
    granularity:str, # Granularity of sparsification (e.g., 'weight', 'filter')
    context:str, # Context for sparsification ('global' or 'local')
    criteria:Criteria, # Criteria to determine which weights to keep
    nm:bool=False, # Whether to use N:M sparsity pattern (forces 2:4 sparsity)
    layer_type:Type[nn.Module]=<class 'torch.nn.modules.conv.Conv2d'>, # Type of layers to apply sparsification to
):

Class providing sparsifying capabilities

The Sparsifier class allows us to remove some weights, that are considered to be less useful than others. This can be done by first creating an instance of the class, specifying:

  • The granularity, i.e. the part of filters that you want to remove. Typically, we usually remove weights, vectors, kernels or even complete filters.
  • The context, i.e. if you want to consider each layer independently (local), or compare the parameters to remove across the whole network (global).
  • The criteria, i.e. the way to assess the usefulness of a parameter. Common methods compare parameters using their magnitude, the lowest magnitude ones considered to be less useful.

Key Methods

User can pass a single layer to sparsify by using the Sparsifier.sparsify_layer method.

source

Sparsifier.sparsify_layer


def sparsify_layer(
    m:nn.Module, # The layer to sparsify
    sparsity:float, # Target sparsity level (percentage)
    round_to:Optional[int]=None, # Round to a multiple of this value
)->None:

Apply sparsification to a single layer

Most of the time, we may want to sparsify the whole model at once, using the Sparsifier.sparsify_model method, indicating the percentage of sparsity you want to apply.

source

Sparsifier.sparsify_model


def sparsify_model(
    sparsity:Union[float, list[float]], # Target sparsity level(s)
    round_to:Optional[int]=None, # Round to a multiple of this value
)->None:

Apply sparsification to all matching layers in the model

Advanced Options

In some case, you may want to impose the remaining amount of parameters to be a multiple of a given number (e.g. 8), this can be done by passing the round_to parameter.

Also, instead of passing a single value of sparsity, a list of sparsities can also be provided. In that case, each value in the list is the sparsity that will be applied to the corresponding layer.

Example: I have a 4-layer network and want to remove half of the parameters from layers 2 and 3, I can provide the list: sparsity = [0, 50, 50, 0]

See Also

  • SparsifyCallback - Apply sparsification during fastai training
  • Criteria - Different importance measures for selecting what to sparsify
  • Granularity - Control what gets sparsified (weights, filters, etc.)
  • Schedules - Control sparsification progression during training
  • Pruner - Structured pruning that removes filters entirely