FakeQuantizer

Round a model onto any weight and activation width, in floating point

Overview

FakeQuantizer rounds a model’s weights and activations onto the grids of the widths you ask for, and leaves everything in floating point. Nothing is packed, no kernel is swapped, no backend is involved: the model still runs its ordinary nn.Conv2d and nn.Linear modules, at its original dtype, on the device it was trained on. What it answers is one question — what does this width cost in accuracy? — before spending a deployment flow to find out.

That is what makes any pair askable. W4A8, W2A8 and W8A4 are all arithmetic here, where a backend table only lists the pairs somebody shipped a kernel for. When you have chosen a width and want a model that actually runs at it, that is [Quantizer](https://FasterAI-Labs.github.io/fasterai/quantize/quantizer.html#quantizer)’s job.

A few things this engine deliberately does not do:

Biases stay in floating point. A deployed integer kernel accumulates them at int32; here they are untouched, so a width claim from this engine is about the weights and activations only.
BatchNorm is not folded. A deployed flow folds it before quantizing, which changes the per-channel weight ranges; fold it yourself first with [BN_Folder](https://FasterAI-Labs.github.io/fasterai/misc/bn_folding.html#bn_folder) if you want the ranges a folded graph would see.
nn.Embedding and nn.ConvTranspose2d are outside the default layer_type. A model holding them keeps those tensors byte-identical while print_precision() still names a width, so a width reported on a transformer does not cover its largest tensor unless you name that type yourself.

The arithmetic

The rounding wraps torch.fake_quantize_per_tensor_affine and torch.fake_quantize_per_channel_affine, which accept any width from 2 to 16 bits on CPU and on CUDA. per_group reshapes each row into groups with einops and reuses the per-channel operator, refusing a group_size that does not divide the row.

Torch’s own MinMaxObserver._calculate_qparams is not reused for the same reason a backend table cannot express W4A8: an observer is bound to a real quantized dtype, so widths such as 2, 6 or 13 have no torch.qint* to be calculated against. Computing the scale here is what makes every width sayable.

A grid always contains 0: the range is widened to include it, and the scale is clamped to an epsilon so a zero-range channel cannot produce a zero scale. One consequence worth knowing — a weight that is exactly 0 quantizes to the zero-point and dequantizes to exactly 0, so sparsity survives the rounding, symmetric or affine.

Calibration reads the fastai interface and nothing else: a DataLoaders, or one of the loaders it holds. A torch DataLoader, a list of batches or a bare tensor is refused by name, pointing at dls or dls.train — fasterai trains and validates through fastai, and calibration is a validation pass.

Only a DataLoaders is unwrapped to its training loader, and it is recognised by carrying loaders, not by answering to .train: a fastai loader delegates an unknown .train to its dataset, which yields unbatched items, so hasattr(dl, 'train') is True and reading it would calibrate on the wrong shapes. Narrowing the accepted types does not remove that trap, because it lives inside fastai.

The activation scales live as buffers on the module that produces them, and _ActRounder is a picklable class rather than a closure, because that hook stays installed — copy.deepcopy treats a closure as atomic, so a copy would share the original’s calibration. The observation hooks below may be closures: they are removed in a finally before anyone can copy the model.

Provenance

The widths a quantizer was asked for are available as fq.spec, and are attached to every model it rounds: fake_quant_spec(model) reads them back. FakeQuantSpec lives beside QuantSpec in Precision, so asking “what precision is this model at?” needs no quantizer import — that page also lists the three places where its vocabulary deliberately diverges from the deployed one. It records what was asked for; it says nothing about what a runtime would do with it.

FakeQuantizer

The arguments name the four things a rounding grid needs:

weight_bits / act_bits: the widths. None leaves that tensor in floating point — a 16 is a real 16-bit grid, not a way of saying “float”.
qscheme: the axis the weight scales are computed along, 'per_tensor', 'per_channel' or 'per_group'. This is the scale axis, not the granularity a Sparsifier removes along. Activations are always rounded per tensor.
symmetric: a zero-point of 0, or an affine offset.
observer: 'static' freezes the activation scales measured by calibrate, 'dynamic' recomputes them on every batch. Unlike [Quantizer](https://FasterAI-Labs.github.io/fasterai/quantize/quantizer.html#quantizer)’s method=, which names a whole flow, this names only how the activation scales are obtained.

Per-layer widths follow the siblings’ form — a dict against the default carried by weight_bits / act_bits, keyed by layer name or by module, warning on a key the quantizer does not round:

FakeQuantizer(model, weight_bits=8, layer_bits={'layer4.1.conv2': 4, 'fc': None})

Usage

Static activations, the ordinary case — measure the ranges, then round:

from fasterai.quantize.fake_quantizer import FakeQuantizer

fq = FakeQuantizer(model, weight_bits=8, act_bits=8)
fq.calibrate(dls)          # observe the activation ranges (a `DataLoaders`, or `dls.train`)
fq.quantize_model()        # round in place, and return the same model
learn.validate()           # what W8A8 costs on your metric
fq.remove()                # bit-identical floating-point weights back

Weight-only, with no calibration to do. group_size has to divide every row it rounds, and a convolution’s row is in_channels * kH * kW — a ResNet’s first convolution is 147 weights wide, which no power of two divides, so name a size that fits or leave that layer out:

FakeQuantizer(model, weight_bits=4, qscheme='per_group', group_size=64,
              layer_bits={'conv1': None}).quantize_model()

Dynamic activations, when you would rather not carry calibration data:

FakeQuantizer(model, 8, 8, observer='dynamic').quantize_model()

print_precision() lists the width every layer was rounded to. It reports widths and nothing else: a rounded model holds a floating-point copy of every weight it rounds, so it takes more memory while rounded, not less, and runs a little slower because of the extra rounding. Those copies are non-persistent buffers — the state dict keeps exactly the keys it had — and remove() frees them.

Overview

The arithmetic

Provenance

FakeQuantizer

Usage

See Also