Quick Start

Sparsify a model while it trains, in five cells

fasterai plugs its compression techniques into a fastai Learner as callbacks, so a model is compressed while it trains rather than after.

pip install fasterai

from fastai.vision.all import *
from fasterai.sparse.all import *

A ResNet-18 on the PETS cat/dog task, images resized to 64 px:

path = untar_data(URLs.PETS)
files = get_image_files(path/"images")

def label_func(f): return f[0].isupper()

dls = ImageDataLoaders.from_name_func(path, files, label_func, item_tfms=Resize(64))
learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()

SparsifyCallback zeroes half of the convolution weights over the fit, layer by layer, keeping the largest magnitudes. It prints the sparsity reached at the end of each epoch, then a per-layer report.

sp_cb = SparsifyCallback(sparsity=0.5, granularity='weight', context='local',
                         criteria=large_final, schedule=one_cycle)
learn.fit_one_cycle(3, cbs=sp_cb)

Sparsifying weight until a sparsity of 50.00%
Saving Weights at epoch 0

epoch	train_loss	valid_loss	accuracy	time
0	0.618441	0.660923	0.841678	00:06
1	0.332341	0.222465	0.905954	00:08
2	0.175007	0.205939	0.918133	00:07

Sparsity at the end of epoch 0: 10.40%
Sparsity at the end of epoch 1: 48.30%
Sparsity at the end of epoch 2: 50.00%
Final Sparsity: 50.00%

Sparsity Report:
--------------------------------------------------------------------------------
Layer                          Type            Params     Zeros      Sparsity  
--------------------------------------------------------------------------------
0.0                            Conv2d          9,408      4,702         49.98%
0.4.0.conv1                    Conv2d          36,864     18,430        49.99%
0.4.0.conv2                    Conv2d          36,864     18,430        49.99%
0.4.1.conv1                    Conv2d          36,864     18,430        49.99%
0.4.1.conv2                    Conv2d          36,864     18,430        49.99%
0.5.0.conv1                    Conv2d          73,728     36,862        50.00%
0.5.0.conv2                    Conv2d          147,456    73,725        50.00%
0.5.0.downsample.0             Conv2d          8,192      4,094         49.98%
0.5.1.conv1                    Conv2d          147,456    73,725        50.00%
0.5.1.conv2                    Conv2d          147,456    73,725        50.00%
0.6.0.conv1                    Conv2d          294,912    147,452       50.00%
0.6.0.conv2                    Conv2d          589,824    294,905       50.00%
0.6.0.downsample.0             Conv2d          32,768     16,382        49.99%
0.6.1.conv1                    Conv2d          589,824    294,905       50.00%
0.6.1.conv2                    Conv2d          589,824    294,905       50.00%
0.7.0.conv1                    Conv2d          1,179,648  589,811       50.00%
0.7.0.conv2                    Conv2d          2,359,296  1,179,624     50.00%
0.7.0.downsample.0             Conv2d          131,072    65,533        50.00%
0.7.1.conv1                    Conv2d          2,359,296  1,179,624     50.00%
0.7.1.conv2                    Conv2d          2,359,296  1,179,624     50.00%
--------------------------------------------------------------------------------
Overall                        all             11,166,912 5,583,318     50.00%

Accuracy on the 1478 validation images, with its Wilson 95% interval (single run):

from math import sqrt

def report(learn, name):
    "Validation accuracy with its Wilson 95% interval"
    n = len(learn.dls.valid_ds)
    with learn.no_bar(): acc = float(learn.validate()[1])
    k, z = round(acc*n), 1.96
    p, d = k/n, 1 + z**2/n
    c = p + z**2/(2*n)
    h = z*sqrt(p*(1-p)/n + z**2/(4*n**2))
    print(f"{name}: {acc:.2%} ({k}/{n}), Wilson 95% [{(c-h)/d:.2%}, {(c+h)/d:.2%}]")

report(learn, "50% sparse")

50% sparse: 91.81% (1357/1478), Wilson 95% [90.30%, 93.10%]

Summary

Tool	What it gives you
`SparsifyCallback(sparsity, granularity, context, criteria, schedule)`	Weights zeroed during training, following the schedule
`Sparsifier(model, granularity, context, criteria)`	The same zeroing, outside a training loop

Summary

See Also