model = resnet18()Sparsifier
Make your neural network sparse with fastai
A sparse vector, as opposed to a dense one, is a vector which contains a lot of zeroes. When we speak about making a neural network sparse, we thus mean that the network’s weights are mostly zeroes.
With fasterai, you can do that thanks to the Sparsifier class.
Let’s start by creating a model
As you probably know, weights in a convolutional neural network have 4 dimensions (\(c_{out} \times c_{in} \times k_h \times k_w\))
model.conv1.weight.ndim4In the case of ResNet18, the dimension of the first layer weights is \(64 \times 3 \times 7 \times 7\). We thus can plot each of the \(64\) filters as a \(7 \times 7\) color image (because they contain \(3\) channels).
plot_kernels(model.conv1)
The Sparsifier class allows us to remove some (part of) the filters, that are considered to be less useful than others. This can be done by first creating an instance of the class, specifying:
- The
granularity, i.e. the part of filters that you want to remove. Typically, we usually remove weights, columns, kernels or even complete filters. - The
context, i.e. if you want to consider each layer independently (local), or compare the parameters to remove across the whole network (global). - The
criteria, i.e. the way to assess the usefulness of a parameter. Common methods compare parameters using their magnitude, the lowest magnitude ones considered to be less useful.
You can pass a single layer to sparsify by using the Sparsifier.sparsify_layer method.
Sparsifier.sparsify_layer
def sparsify_layer(
m:nn.Module, # The layer to sparsify
sparsity:float, # Target sparsity, a fraction in [0, 1] (0.4 = 40%)
round_to:int | None=None, # Round to a multiple of this value
)->None:
Apply sparsification to a single layer
model = resnet18()
sparsifier = Sparsifier(model, 'filter', 'local', large_final)
sparsifier.sparsify_layer(model.conv1, 0.7)sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 6,321 67.19%
layer1.0.conv1 Conv2d 36,864 0 0.00%
layer1.0.conv2 Conv2d 36,864 0 0.00%
layer1.1.conv1 Conv2d 36,864 0 0.00%
layer1.1.conv2 Conv2d 36,864 0 0.00%
layer2.0.conv1 Conv2d 73,728 0 0.00%
layer2.0.conv2 Conv2d 147,456 0 0.00%
layer2.0.downsample.0 Conv2d 8,192 0 0.00%
layer2.1.conv1 Conv2d 147,456 0 0.00%
layer2.1.conv2 Conv2d 147,456 0 0.00%
layer3.0.conv1 Conv2d 294,912 0 0.00%
layer3.0.conv2 Conv2d 589,824 0 0.00%
layer3.0.downsample.0 Conv2d 32,768 0 0.00%
layer3.1.conv1 Conv2d 589,824 0 0.00%
layer3.1.conv2 Conv2d 589,824 0 0.00%
layer4.0.conv1 Conv2d 1,179,648 1 0.00%
layer4.0.conv2 Conv2d 2,359,296 0 0.00%
layer4.0.downsample.0 Conv2d 131,072 0 0.00%
layer4.1.conv1 Conv2d 2,359,296 0 0.00%
layer4.1.conv2 Conv2d 2,359,296 0 0.00%
--------------------------------------------------------------------------------
Overall all 11,166,912 6,322 0.06%Only conv1 moved: 6,321 of its 9,408 weights are zero, i.e. 43 of its 64 filters. That is 67.19%, not the 70% asked for — with granularity='filter' the count of removed filters is truncated, and here it lands one filter below int(0.7 * 64) = 44.
Most of the time, we may want to sparsify the whole model at once, using the Sparsifier.sparsify_model method, indicating the sparsity you want to apply as a fraction in \([0, 1]\).
Sparsifier.sparsify_model
def sparsify_model(
sparsity:float | dict, # Target sparsity, a fraction in [0, 1] (0.4 = 40%), or a per-layer dict
round_to:int | None=None, # Round to a multiple of this value
)->None:
Apply sparsification to all matching layers in the model
There are several ways in which we can make that first layer sparse. You will find the most important below:
model = resnet18()
sparsifier = Sparsifier(model, 'weight', 'local', large_final)
sparsifier.sparsify_model(0.7)sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 6,584 69.98%
layer1.0.conv1 Conv2d 36,864 25,803 70.00%
layer1.0.conv2 Conv2d 36,864 25,803 70.00%
layer1.1.conv1 Conv2d 36,864 25,803 70.00%
layer1.1.conv2 Conv2d 36,864 25,803 70.00%
layer2.0.conv1 Conv2d 73,728 51,608 70.00%
layer2.0.conv2 Conv2d 147,456 103,218 70.00%
layer2.0.downsample.0 Conv2d 8,192 5,733 69.98%
layer2.1.conv1 Conv2d 147,456 103,218 70.00%
layer2.1.conv2 Conv2d 147,456 103,218 70.00%
layer3.0.conv1 Conv2d 294,912 206,437 70.00%
layer3.0.conv2 Conv2d 589,824 412,875 70.00%
layer3.0.downsample.0 Conv2d 32,768 22,936 70.00%
layer3.1.conv1 Conv2d 589,824 412,875 70.00%
layer3.1.conv2 Conv2d 589,824 412,875 70.00%
layer4.0.conv1 Conv2d 1,179,648 825,752 70.00%
layer4.0.conv2 Conv2d 2,359,296 1,651,506 70.00%
layer4.0.downsample.0 Conv2d 131,072 91,749 70.00%
layer4.1.conv1 Conv2d 2,359,296 1,651,506 70.00%
layer4.1.conv2 Conv2d 2,359,296 1,651,506 70.00%
--------------------------------------------------------------------------------
Overall all 11,166,912 7,816,808 70.00%You now have a model that is \(70\%\) sparse !
Granularity
As we said earlier, the granularity defines the structure of parameter that you will remove.
In the example below, we removed weight from each convolutional filter, meaning that we now have sparse filters, as can be seen in the image below:
plot_kernels(model.conv1)
Another granularity is, for example, removing column vectors from the filters. To do so, just change the granularity parameter accordingly.
model = resnet18()
sparsifier = Sparsifier(model, 'column', 'local', large_final)
sparsifier.sparsify_layer(model.conv1, 0.7)plot_kernels(model.conv1)
For more information and examples about the sparsification granularities, take a look at the Granularity page.
Context
The context defines where to look in the model, i.e. from where we compare the weights. The two basic contexts are: * local, i.e. we compare the weights of each layer individually. This will lead to layers with similar levels of sparsity. * global, i.e. we compare the weights of the whole model. This will lead to layers with different levels of sparsity.
model = resnet18()
sparsifier = Sparsifier(model, 'weight', 'global', large_final)
sparsifier.sparsify_model(0.7)sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 6,301 66.97%
layer1.0.conv1 Conv2d 36,864 11,907 32.30%
layer1.0.conv2 Conv2d 36,864 11,894 32.26%
layer1.1.conv1 Conv2d 36,864 11,816 32.05%
layer1.1.conv2 Conv2d 36,864 11,803 32.02%
layer2.0.conv1 Conv2d 73,728 32,454 44.02%
layer2.0.conv2 Conv2d 147,456 64,971 44.06%
layer2.0.downsample.0 Conv2d 8,192 1,244 15.19%
layer2.1.conv1 Conv2d 147,456 65,377 44.34%
layer2.1.conv2 Conv2d 147,456 65,098 44.15%
layer3.0.conv1 Conv2d 294,912 175,050 59.36%
layer3.0.conv2 Conv2d 589,824 349,742 59.30%
layer3.0.downsample.0 Conv2d 32,768 7,031 21.46%
layer3.1.conv1 Conv2d 589,824 349,196 59.20%
layer3.1.conv2 Conv2d 589,824 349,824 59.31%
layer4.0.conv1 Conv2d 1,179,648 894,692 75.84%
layer4.0.conv2 Conv2d 2,359,296 1,789,065 75.83%
layer4.0.downsample.0 Conv2d 131,072 40,062 30.56%
layer4.1.conv1 Conv2d 2,359,296 1,790,317 75.88%
layer4.1.conv2 Conv2d 2,359,296 1,788,993 75.83%
--------------------------------------------------------------------------------
Overall all 11,166,912 7,816,837 70.00%With global, the threshold is computed over the whole network at once. The overall sparsity is still 70.00%, but the per-layer values now spread from 15.19% to 75.88%, whereas the local report above put every layer at 70.00%.
Criteria
The criteria defines how we select the parameters to remove. It is usually given by a scoring method. The most common one is large_final, i.e. keep the parameters with the highest absolute value, as they are supposed to contribute the most to the final results of the model. Below we run its opposite, small_final, on the same global setting as the report above.
model = resnet18()
sparsifier = Sparsifier(model, 'weight', 'global', small_final)
sparsifier.sparsify_model(0.7)sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 9,360 99.49%
layer1.0.conv1 Conv2d 36,864 614 1.67%
layer1.0.conv2 Conv2d 36,864 260 0.71%
layer1.1.conv1 Conv2d 36,864 284 0.77%
layer1.1.conv2 Conv2d 36,864 149 0.40%
layer2.0.conv1 Conv2d 73,728 4,281 5.81%
layer2.0.conv2 Conv2d 147,456 4,008 2.72%
layer2.0.downsample.0 Conv2d 8,192 8 0.10%
layer2.1.conv1 Conv2d 147,456 8,331 5.65%
layer2.1.conv2 Conv2d 147,456 5,407 3.67%
layer3.0.conv1 Conv2d 294,912 96,244 32.63%
layer3.0.conv2 Conv2d 589,824 234,603 39.78%
layer3.0.downsample.0 Conv2d 32,768 86 0.26%
layer3.1.conv1 Conv2d 589,824 80,673 13.68%
layer3.1.conv2 Conv2d 589,824 160,316 27.18%
layer4.0.conv1 Conv2d 1,179,648 1,179,647 100.00%
layer4.0.conv2 Conv2d 2,359,296 1,591,176 67.44%
layer4.0.downsample.0 Conv2d 131,072 822 0.63%
layer4.1.conv1 Conv2d 2,359,296 2,084,800 88.37%
layer4.1.conv2 Conv2d 2,359,296 2,355,767 99.85%
--------------------------------------------------------------------------------
Overall all 11,166,912 7,816,836 70.00%small_final scores the smallest magnitudes as the most important, so it zeroes the weights large_final would have kept: conv1 ends at 99.49% zeros, while layer1.0.conv1 loses only 614 of its 36,864 weights.
For more information and examples about the sparsification criteria, take a look at the Criteria page.
Per-layer targets
sparsify_model also accepts a dict mapping layer names — those printed in the report — to their own sparsity. Layers absent from the dict are left untouched, and a dict requires context='local'.
model = resnet18()
sparsifier = Sparsifier(model, 'weight', 'local', large_final)
sparsifier.sparsify_model({'layer1.0.conv1': 0.3, 'layer4.1.conv2': 0.8})
sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 0 0.00%
layer1.0.conv1 Conv2d 36,864 11,058 30.00%
layer1.0.conv2 Conv2d 36,864 0 0.00%
layer1.1.conv1 Conv2d 36,864 0 0.00%
layer1.1.conv2 Conv2d 36,864 0 0.00%
layer2.0.conv1 Conv2d 73,728 0 0.00%
layer2.0.conv2 Conv2d 147,456 0 0.00%
layer2.0.downsample.0 Conv2d 8,192 0 0.00%
layer2.1.conv1 Conv2d 147,456 0 0.00%
layer2.1.conv2 Conv2d 147,456 0 0.00%
layer3.0.conv1 Conv2d 294,912 0 0.00%
layer3.0.conv2 Conv2d 589,824 0 0.00%
layer3.0.downsample.0 Conv2d 32,768 0 0.00%
layer3.1.conv1 Conv2d 589,824 0 0.00%
layer3.1.conv2 Conv2d 589,824 0 0.00%
layer4.0.conv1 Conv2d 1,179,648 0 0.00%
layer4.0.conv2 Conv2d 2,359,296 0 0.00%
layer4.0.downsample.0 Conv2d 131,072 0 0.00%
layer4.1.conv1 Conv2d 2,359,296 0 0.00%
layer4.1.conv2 Conv2d 2,359,296 1,887,435 80.00%
--------------------------------------------------------------------------------
Overall all 11,166,912 1,898,493 17.00%Only the two declared layers moved, to 30.00% and 80.00%; the other 18 stay at 0.00%. The overall figure is parameter-weighted, so it reads 17.00%.
Remark
In some cases, you may want to impose the remaining amount of parameters to be a multiple of 8, this can be done by passing the round_to parameter.
model = resnet18()
sparsifier = Sparsifier(model, 'filter', 'local', large_final)
sparsifier.sparsify_model(0.7, round_to=8)sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 5,880 62.50%
layer1.0.conv1 Conv2d 36,864 23,040 62.50%
layer1.0.conv2 Conv2d 36,864 23,040 62.50%
layer1.1.conv1 Conv2d 36,864 23,040 62.50%
layer1.1.conv2 Conv2d 36,864 23,040 62.50%
layer2.0.conv1 Conv2d 73,728 50,688 68.75%
layer2.0.conv2 Conv2d 147,456 101,376 68.75%
layer2.0.downsample.0 Conv2d 8,192 5,632 68.75%
layer2.1.conv1 Conv2d 147,456 101,376 68.75%
layer2.1.conv2 Conv2d 147,456 101,376 68.75%
layer3.0.conv1 Conv2d 294,912 202,752 68.75%
layer3.0.conv2 Conv2d 589,824 405,504 68.75%
layer3.0.downsample.0 Conv2d 32,768 22,528 68.75%
layer3.1.conv1 Conv2d 589,824 405,504 68.75%
layer3.1.conv2 Conv2d 589,824 405,504 68.75%
layer4.0.conv1 Conv2d 1,179,648 811,008 68.75%
layer4.0.conv2 Conv2d 2,359,296 1,622,016 68.75%
layer4.0.downsample.0 Conv2d 131,072 90,112 68.75%
layer4.1.conv1 Conv2d 2,359,296 1,622,016 68.75%
layer4.1.conv2 Conv2d 2,359,296 1,622,016 68.75%
--------------------------------------------------------------------------------
Overall all 11,166,912 7,667,448 68.66%With local, round_to=8 applies per layer on top of a per-layer 0.7: every layer lands on a multiple of 8 kept filters, at 62.50% or 68.75%, for 68.66% overall.
model = resnet18()
sparsifier = Sparsifier(model, 'filter', 'global', large_final)
sparsifier.sparsify_model(0.7, round_to=8)sparsifier.print_sparsity()
Sparsity Report:
--------------------------------------------------------------------------------
Layer Type Params Zeros Sparsity
--------------------------------------------------------------------------------
conv1 Conv2d 9,408 8,232 87.50%
layer1.0.conv1 Conv2d 36,864 0 0.00%
layer1.0.conv2 Conv2d 36,864 0 0.00%
layer1.1.conv1 Conv2d 36,864 0 0.00%
layer1.1.conv2 Conv2d 36,864 0 0.00%
layer2.0.conv1 Conv2d 73,728 69,120 93.75%
layer2.0.conv2 Conv2d 147,456 138,240 93.75%
layer2.0.downsample.0 Conv2d 8,192 0 0.00%
layer2.1.conv1 Conv2d 147,456 138,240 93.75%
layer2.1.conv2 Conv2d 147,456 129,024 87.50%
layer3.0.conv1 Conv2d 294,912 285,696 96.88%
layer3.0.conv2 Conv2d 589,824 571,392 96.88%
layer3.0.downsample.0 Conv2d 32,768 0 0.00%
layer3.1.conv1 Conv2d 589,824 571,392 96.88%
layer3.1.conv2 Conv2d 589,824 552,960 93.75%
layer4.0.conv1 Conv2d 1,179,648 1,161,216 98.44%
layer4.0.conv2 Conv2d 2,359,296 2,322,432 98.44%
layer4.0.downsample.0 Conv2d 131,072 0 0.00%
layer4.1.conv1 Conv2d 2,359,296 2,322,432 98.44%
layer4.1.conv2 Conv2d 2,359,296 2,322,432 98.44%
--------------------------------------------------------------------------------
Overall all 11,166,912 10,592,808 94.86%With global, the rounding happens per layer after a single network-wide threshold. This 0.7 request lands at 94.86% overall and leaves seven layers untouched, so read the report when combining round_to with global.
Summary
| Tool | Purpose |
|---|---|
Sparsifier |
Core class for zeroing out weights |
sparsify_model() |
Apply sparsification to all matching layers, uniformly or per layer |
sparsify_layer() |
Apply sparsification to a single layer |
print_sparsity() |
Report per-layer and overall sparsity |
large_final / small_final |
Criteria: keep the largest / the smallest magnitudes |
| Granularity options | weight, column, row, kernel, filter, … |
| Context options | local (per-layer) vs global (network-wide) |
round_to |
Round the number of kept parameters per layer |
See Also
- Sparsifier API - Full Sparsifier API reference
- SparsifyCallback - Integrate sparsification into training
- Schedules - Control sparsification progression
- Criteria - All importance scoring methods
- Granularity - What gets sparsified at each level