Overview

Features • Installation • Tutorials • Citing • License

fasterai is a library created to make neural network smaller and faster. It gathers the common compression techniques — sparsification, pruning, regularization, knowledge distillation, quantization — and the steps that follow them: exporting the compressed model, and measuring which layers can take the compression.

Each technique is built around the same four modules: granularity, context, criteria, schedule. Each of them is customizable, so you can change them according to your needs or come up with your own.

Project Documentation

Visit the Read The Docs Project Page or read the following README to know more about using fasterai.

Features

1. Sparsifying

Make your model sparse according to a:
- Sparsity: the fraction of weights that will be replaced by 0
- Granularity: the granularity at which you operate the sparsification (weights, vectors, kernels, filters)
- Context: sparsify either each layer independently (local) or the whole model (global)
- Criteria: the criteria used to select the weights to remove (magnitude, movement, …)
- Schedule: which schedule you want to follow (one shot, iterative, gradual, …) 

SparsifyCallback(sparsity=0.5, granularity='weight', context='local',
                 criteria=large_final, schedule=one_cycle)

2. Pruning

Once your model has useless nodes due to zero-weights, they can be removed to not be a part of the network anymore. The model that comes out is structurally smaller.

PruneCallback(pruning_ratio=0.3, schedule=one_cycle, context='global', criteria=large_final)

3. Regularization

Instead of explicitly making your network sparse, let it train towards sparse connections by pushing the weights of a group to be as small as possible. Regularization follows the same granularities as sparsifying.

RegularizeCallback(criteria=large_final, granularity='filter', weight=0.01)

4. Knowledge Distillation

alt text

Distill the knowledge acquired by a big model into a smaller one, comparing their predictions or their intermediate activations.

KnowledgeDistillationCallback(teacher, loss=SoftTarget, weight=0.5)

5. Lottery Ticket Hypothesis

Find the winning ticket in your network, i.e. re-train a sparse subnetwork from the initial weights it started with.

SparsifyCallback(sparsity=0.5, granularity='weight', context='local', criteria=large_final,
                 schedule=iterative, lth=True, rewind_epoch=1)

6. Quantization

Lower the arithmetic to INT8, either after training on calibration data or during it. The precision is named argument by argument, and a backend that cannot honor it refuses instead of quantizing something else.

Quantizer(backend='pt2e', method='static', weight_bits=8, act_bits=8, qscheme='per_channel',
          symmetric=True).quantize(model, calibration_dl)

7. Export

Write the compressed model as an ONNX file, then read back what the exporter produced rather than what it was asked for.

path = export_qdq(qmodel, sample, 'model.onnx')
qdq_stats(path)                       # Q/DQ nodes, per-channel scales, zero-points
verify_qdq(qmodel, path, samples)     # argmax agreement with PyTorch

8. Architecture rewrites

Fold batch norm into the preceding convolution, factorize fully-connected or convolution layers, or prepare a model for CPU inference.

BN_Folder().fold(model)
FC_Decomposer().decompose(model)
Conv_Decomposer().decompose(model, method='tucker')
optimize_for_cpu(model, sample)

The FasterAI Ecosystem

fasterai is part of a family of libraries designed to make neural network optimization accessible:

Package Purpose Scope
fasterai Compression techniques Pruning, sparsification, distillation, quantization during training
fasterbench Benchmarking Measuring model size, speed, memory, compute, and energy
fasterlatency Latency prediction Hardware-aware neural architecture search

Typical Workflow

  1. Benchmark your model with fasterbench to identify bottlenecks
  2. Compress using fasterai techniques (pruning, distillation, quantization)
  3. Validate compression impact with fasterbench
  4. Deploy the optimized model

Quick Start

0. Import fasterai

from fasterai.sparse.all import *

1. Create your model with fastai

learn = vision_learner(dls, resnet18, metrics=accuracy)

2. Get your fasterai callback

sp_cb = SparsifyCallback(sparsity=0.5, granularity='weight', context='local',
                         criteria=large_final, schedule=one_cycle)

3. Train your model to make it sparse !

learn.fit_one_cycle(3, cbs=sp_cb)

Compression ratios are fractions in [0, 1]: sparsity=0.5 zeroes half of the weights.

Installation

pip install git+https://github.com/FasterAI-Labs/fasterai.git

or

pip install fasterai

Tutorials

Citing

@software{Hubens,
  author       = {Nathan Hubens},
  title        = {fasterai},
  year         = 2022,
  publisher    = {Zenodo},
  version      = {v0.3.3},
  doi          = {10.5281/zenodo.6469868},
  url          = {https://doi.org/10.5281/zenodo.6469868}
}

License

Apache-2.0 License.