Sparsify Callback

Use the sparsifier in fastai Callback system 
from fastai.vision.all import *
from fasterai.sparse.all import *

SparsifyCallback applies a Sparsifier during training: at each step it computes the current sparsity from a Schedule and zeroes the weights accordingly.

Get your data

path = untar_data(URLs.PETS)
files = get_image_files(path/"images")

def label_func(f): return f[0].isupper()

dls = ImageDataLoaders.from_name_func(path, files, label_func, item_tfms=Resize(64))

The task is binary (cat/dog). Every accuracy below is read against this split’s size and majority-class rate.

def accuracy_report(learn, z=1.96):
    "Validation accuracy as k/n, with its Wilson 95% interval"
    n = len(learn.dls.valid_ds); k = round(learn.validate()[1]*n); p, d = k/n, 1 + z**2/n
    c, h = (p + z**2/(2*n))/d, z*(p*(1-p)/n + z**2/(4*n**2))**0.5/d
    print(f'{k}/{n} = {p:.2%}, Wilson 95% [{c-h:.2%}, {c+h:.2%}]')
    return k

labels = [label_func(f.name) for f in dls.valid_ds.items]
print(f'validation set: n={len(labels)}, majority class {max(sum(labels), len(labels)-sum(labels))/len(labels):.2%}')
validation set: n=1478, majority class 65.83%

Train a dense model as a baseline

learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()

learn.fit_one_cycle(5)
epoch train_loss valid_loss accuracy time
0 0.709514 0.702051 0.809202 00:03
1 0.406462 0.304271 0.882950 00:03
2 0.243738 0.208367 0.912720 00:03
3 0.135865 0.173228 0.929635 00:03
4 0.070623 0.170489 0.937077 00:03 
k_dense = accuracy_report(learn)
1385/1478 = 93.71%, Wilson 95% [92.35%, 94.84%]

Let’s now try adding some sparsity in our model

learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()

The SparsifyCallback takes the same granularity, context and criteria as the Sparsifier, plus a schedule, which says how the sparsity grows during training.

The schedule is a fasterai Schedule object (one_cycle, cos, lin, one_shot, iterative, agp, dsd); to use a fastai annealing function, wrap it: Schedule(sched_cos).

sp_cb = SparsifyCallback(sparsity=0.5, granularity='weight', context='local', criteria=large_final, schedule=one_cycle)
learn.fit_one_cycle(5, cbs=sp_cb)
Sparsifying weight until a sparsity of 50.00%
Saving Weights at epoch 0
epoch train_loss valid_loss accuracy time
0 0.669916 0.495064 0.839648 00:06
1 0.380538 0.334417 0.865359 00:06
2 0.233458 0.240748 0.899188 00:05
3 0.116518 0.184529 0.934371 00:05
4 0.062299 0.186647 0.936401 00:05 
Sparsity at the end of epoch 0: 1.96%
Sparsity at the end of epoch 1: 20.07%
Sparsity at the end of epoch 2: 45.86%
Sparsity at the end of epoch 3: 49.74%
Sparsity at the end of epoch 4: 50.00%
Final Sparsity: 50.00%

Sparsity Report:
--------------------------------------------------------------------------------
Layer                          Type            Params     Zeros      Sparsity  
--------------------------------------------------------------------------------
0.0                            Conv2d          9,408      4,702         49.98%
0.4.0.conv1                    Conv2d          36,864     18,430        49.99%
0.4.0.conv2                    Conv2d          36,864     18,430        49.99%
0.4.1.conv1                    Conv2d          36,864     18,430        49.99%
0.4.1.conv2                    Conv2d          36,864     18,430        49.99%
0.5.0.conv1                    Conv2d          73,728     36,862        50.00%
0.5.0.conv2                    Conv2d          147,456    73,726        50.00%
0.5.0.downsample.0             Conv2d          8,192      4,094         49.98%
0.5.1.conv1                    Conv2d          147,456    73,726        50.00%
0.5.1.conv2                    Conv2d          147,456    73,726        50.00%
0.6.0.conv1                    Conv2d          294,912    147,453       50.00%
0.6.0.conv2                    Conv2d          589,824    294,908       50.00%
0.6.0.downsample.0             Conv2d          32,768     16,382        49.99%
0.6.1.conv1                    Conv2d          589,824    294,908       50.00%
0.6.1.conv2                    Conv2d          589,824    294,908       50.00%
0.7.0.conv1                    Conv2d          1,179,648  589,817       50.00%
0.7.0.conv2                    Conv2d          2,359,296  1,179,635     50.00%
0.7.0.downsample.0             Conv2d          131,072    65,534        50.00%
0.7.1.conv1                    Conv2d          2,359,296  1,179,635     50.00%
0.7.1.conv2                    Conv2d          2,359,296  1,179,635     50.00%
--------------------------------------------------------------------------------
Overall                        all             11,166,912 5,583,371     50.00%
k_sparse = accuracy_report(learn)
1384/1478 = 93.64%, Wilson 95% [92.28%, 94.77%]
print(f'{abs(k_dense - k_sparse)} of {len(dls.valid_ds)} predictions apart')
1 of 1478 predictions apart

Half the weights of every convolution are zero. The two runs above are 1 of 1478 predictions apart, and their Wilson 95% intervals nearly coincide: a single run at this n cannot separate them.

Per-layer targets

sparsity also accepts a dict mapping layer names — those printed in the sparsity report — to their own target. Layers absent from the dict stay dense, and a dict requires context='local'.

per_layer_sparsity = {
    '0.4.0.conv1': 0.3, '0.4.0.conv2': 0.3, '0.4.1.conv1': 0.3, '0.4.1.conv2': 0.3,
    '0.5.0.conv1': 0.5, '0.5.0.conv2': 0.5, '0.5.1.conv1': 0.5, '0.5.1.conv2': 0.5,
    '0.6.0.conv1': 0.7, '0.6.0.conv2': 0.7, '0.6.1.conv1': 0.7, '0.6.1.conv2': 0.7,
    '0.7.0.conv1': 0.8, '0.7.0.conv2': 0.8, '0.7.1.conv1': 0.8, '0.7.1.conv2': 0.8,
}
learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()

sp_cb = SparsifyCallback(sparsity=per_layer_sparsity, granularity='weight', context='local',
                         criteria=large_final, schedule=cos)

learn.fit_one_cycle(5, cbs=sp_cb)
Sparsifying weight until a sparsity of {'0.4.0.conv1': '30.00%', '0.4.0.conv2': '30.00%', '0.4.1.conv1': '30.00%', '0.4.1.conv2': '30.00%', '0.5.0.conv1': '50.00%', '0.5.0.conv2': '50.00%', '0.5.1.conv1': '50.00%', '0.5.1.conv2': '50.00%', '0.6.0.conv1': '70.00%', '0.6.0.conv2': '70.00%', '0.6.1.conv1': '70.00%', '0.6.1.conv2': '70.00%', '0.7.0.conv1': '80.00%', '0.7.0.conv2': '80.00%', '0.7.1.conv1': '80.00%', '0.7.1.conv2': '80.00%'}
Saving Weights at epoch 0
epoch train_loss valid_loss accuracy time
0 0.691505 0.586751 0.824763 00:04
1 0.424129 0.298597 0.872124 00:05
2 0.263313 0.214177 0.909337 00:06
3 0.161415 0.171583 0.933694 00:05
4 0.094364 0.165356 0.933694 00:05 
Sparsity at the end of epoch 0: avg=5.49%
Sparsity at the end of epoch 1: avg=19.87%
Sparsity at the end of epoch 2: avg=37.63%
Sparsity at the end of epoch 3: avg=52.01%
Sparsity at the end of epoch 4: avg=57.50%
Final Sparsity: {'0.4.0.conv1': '30.00%', '0.4.0.conv2': '30.00%', '0.4.1.conv1': '30.00%', '0.4.1.conv2': '30.00%', '0.5.0.conv1': '50.00%', '0.5.0.conv2': '50.00%', '0.5.1.conv1': '50.00%', '0.5.1.conv2': '50.00%', '0.6.0.conv1': '70.00%', '0.6.0.conv2': '70.00%', '0.6.1.conv1': '70.00%', '0.6.1.conv2': '70.00%', '0.7.0.conv1': '80.00%', '0.7.0.conv2': '80.00%', '0.7.1.conv1': '80.00%', '0.7.1.conv2': '80.00%'}

Sparsity Report:
--------------------------------------------------------------------------------
Layer                          Type            Params     Zeros      Sparsity  
--------------------------------------------------------------------------------
0.0                            Conv2d          9,408      0              0.00%
0.4.0.conv1                    Conv2d          36,864     11,058        30.00%
0.4.0.conv2                    Conv2d          36,864     11,058        30.00%
0.4.1.conv1                    Conv2d          36,864     11,058        30.00%
0.4.1.conv2                    Conv2d          36,864     11,058        30.00%
0.5.0.conv1                    Conv2d          73,728     36,862        50.00%
0.5.0.conv2                    Conv2d          147,456    73,726        50.00%
0.5.0.downsample.0             Conv2d          8,192      0              0.00%
0.5.1.conv1                    Conv2d          147,456    73,726        50.00%
0.5.1.conv2                    Conv2d          147,456    73,726        50.00%
0.6.0.conv1                    Conv2d          294,912    206,435       70.00%
0.6.0.conv2                    Conv2d          589,824    412,871       70.00%
0.6.0.downsample.0             Conv2d          32,768     0              0.00%
0.6.1.conv1                    Conv2d          589,824    412,871       70.00%
0.6.1.conv2                    Conv2d          589,824    412,871       70.00%
0.7.0.conv1                    Conv2d          1,179,648  943,708       80.00%
0.7.0.conv2                    Conv2d          2,359,296  1,887,417     80.00%
0.7.0.downsample.0             Conv2d          131,072    0              0.00%
0.7.1.conv1                    Conv2d          2,359,296  1,887,417     80.00%
0.7.1.conv2                    Conv2d          2,359,296  1,887,417     80.00%
--------------------------------------------------------------------------------
Overall                        all             11,166,912 8,353,279     74.80%
accuracy_report(learn);
1380/1478 = 93.37%, Wilson 95% [91.99%, 94.53%]

The declared layers reach exactly their targets and the others stay at 0.00%. The reported overall sparsity is parameter-weighted: the layers carrying 0.7 and 0.8 are also the largest ones. This run changes both the targets and the schedule (cos instead of one_cycle), so its accuracy is not comparable with the run above.

Summary

Tool What it gives you
SparsifyCallback Sparsifies the model during a fastai fit
sparsity A single target, or a dict of layer name → target
granularity / context / criteria The same knobs as the Sparsifier
schedule How the target is reached over the training
Sparsity report Per-layer and overall sparsity, at the end of the fit

See Also