Quantizer

Quantize your network

qdq_placement and skip_activations are both applied at prepare_pt2e time, on the graph torch has just annotated, so the prepared model, the QAT run and the exported file all carry them — which is what lets verify_qdq compare that file with the very model the caller holds.

Overview

The Quantizer class provides model quantization capabilities to reduce model size and improve inference speed. It supports three families of backends: the legacy torch.ao.quantization (FX graph mode), the modern torchao library, and the pt2e (PyTorch 2 Export) flow.

Backend Selection Guide

Backend Bit Widths Target Layers Best For
'x86' INT8 Conv2d + Linear CNN deployment on Intel/AMD CPUs
'qnnpack' INT8 Conv2d + Linear Mobile (ARM) deployment
'torchao' INT8, INT4 Linear (primary) Transformers, MLPs, modern models
'pt2e' INT8 Conv2d + Linear Portable Q/DQ graphs to export to ONNX

Method Selection Guide

Legacy backends (x86, qnnpack, fbgemm, onednn):

Method Needs Calibration When to Use
'static' Yes Best accuracy for CNNs
'dynamic' No Quick experiments, RNNs
'qat' Training Maximum accuracy critical

torchao backend:

Method Needs Calibration When to Use
'int8_weight_only' No General purpose, good default
'int8_dynamic' No Activation + weight quantization

Both recipes rewrite nn.Linear layers only, so a model with more than one convolution, none of them rewritten, is refused rather than handed back with its convolutions in floating point; the refusal names a backend that quantizes them.

pt2e backend:

Method Needs Calibration When to Use
'static' Yes The only method supported in this release

The pt2e backend captures the model with torch.export, then quantizes activations and weights symmetrically, so every quantized tensor has a zero-point of 0. That matters downstream: some ONNX consumers only accept quantized tensors whose zero_point is 0. Export the result with export_qdq to keep the Q/DQ nodes in the ONNX graph.

Activations are observed with a clipping histogram observer, which fits the INT8 grid to the bulk of each tensor instead of to the calibration tail; calibration is slower than it would be with a plain min/max observer. The symmetric activation grid still costs accuracy, and how much of it depends on the architecture — score the quantized model against its float parent before shipping it, and pass symmetric=False to trade the zero-point guarantee for torch’s wider affine activations.

Naming a Precision

method says when quantization happens (before, during or after training). The keyword-only arguments say what precision it produces:

Argument Meaning
weight_bits Weight width — an int, or a {layer_name: width} dict; 16 leaves a layer in floating point
act_bits Activation width; 16 leaves the activations in floating point (weight-only quantization)
qscheme Weight axis: 'per_tensor', 'per_channel' or 'per_group'
group_size Number of weights sharing one scale, with qscheme='per_group'
symmetric True forces a zero-point of 0 on every tensor
qdq_placement Where the Q/DQ pairs sit: 'per_op', or 'skip_conv_add' to leave the residual conv→add edge unquantized ('pt2e' only)
skip_activations Module names whose activations keep the model’s float dtype ('pt2e' only)

Left at None they resolve to what the backend has always done, so existing calls are unchanged. A precision a backend cannot honor raises at construction, naming a backend that can — it is never silently replaced by a nearby one. See Precision for the support matrix, which Quantizer validates against.

A {layer_name: width} dict is honored by the legacy backends with method='static' or 'qat' (over the module types their default mapping rewrites, and over the containers holding them) and by torchao weight-only, act_bits=16 (over the linear layers it rewrites). It is a size lever: the layers named 16 keep their floating-point weights. method='dynamic' quantizes every eligible layer at once and reads no per-module configuration, so it refuses a dict rather than ignoring it.

source

Quantizer

def Quantizer(
    backend:str='x86', # Target backend: 'x86', 'qnnpack', 'fbgemm', 'onednn', 'torchao', or 'pt2e'
    method:str='static', # Method: 'static', 'dynamic', 'qat', 'int8_weight_only', 'int8_dynamic'
    qconfig_mapping:dict | None=None, # Optional custom quantization config (legacy backends only)
    custom_configs:dict | None=None,
    use_per_tensor:bool=False, # Force per-tensor quantization (legacy backends only)
    verbose:bool=False, *,
    weight_bits:int | dict | None=None, # Weight width, or {layer_name: width}; None = the backend's own
    act_bits:int | None=None, # Activation width (16 leaves them in floating point); None = the backend's own
    qscheme:str | None=None, # Weight axis: 'per_tensor', 'per_channel' or 'per_group'; None = the backend's own
    group_size:int | None=None, # Weights sharing one scale (qscheme='per_group')
    symmetric:bool | None=None, # Force a zero-point of 0 everywhere; None = the backend's own
    qdq_placement:str | None=None, # Where the Q/DQ pairs sit: 'per_op', 'skip_conv_add' ('pt2e' only); None = the backend's own
    skip_activations:list[str] | None=None, # Modules whose activations stay in floating point ('pt2e')
):

Initialize a quantizer with specified backend and options.

qdq_placement='skip_conv_add' is opt-in and changes the arithmetic: it leaves the residual branch’s convolution feeding its addition without a Q/DQ pair, so that result arrives at accumulator precision. What a runtime then does with that graph is the runtime’s own property, and the effect on a TRAINED model’s accuracy is UNMEASURED here.

skip_activations=['features.0'] leaves the activations that module produces in floating point while its weights and its input stay INT8; the operator reading them then takes a float input, so what a runtime makes of that graph is UNMEASURED here too.

The resolved precision is available as quantizer.spec, and is attached to every model it quantizes as model._fasterai_quant_spec.

source

Quantizer.quantize

def quantize(
    model:torch.nn.modules.module.Module,
    calibration_dl:Any=None, # Calibration data ('static', 'qat', 'pt2e'); calibrate on non-augmented batches
    max_calibration_samples:int=100, device:str | torch.device='cpu', # Where calibration runs; this flow is CPU-only
)->torch.nn.modules.module.Module:

Quantize a model using the specified backend and method.

Usage Examples

Dynamic Quantization (No calibration needed)

from fasterai.quantize.quantizer import Quantizer

# Create quantizer for dynamic quantization
quantizer = Quantizer(
    backend='x86',
    method='dynamic'
)

# Quantize - no dataloader needed
quantized_model = quantizer.quantize(model, calibration_dl=dls.valid)

Mobile Deployment (ARM devices)

from fasterai.quantize.quantizer import Quantizer

# Use qnnpack backend for mobile
quantizer = Quantizer(
    backend='qnnpack',
    method='static'
)

quantized_model = quantizer.quantize(model, calibration_dl=dls.valid)

Portable INT8 for ONNX Export (pt2e)

from fasterai.quantize.quantizer import Quantizer
from fasterai.export.all import export_qdq, qdq_stats, verify_qdq

# `torch.export` freezes the batch size: calibrate and deploy with the same one
quantizer = Quantizer(backend='pt2e', method='static')
quantized_model = quantizer.quantize(model, calibration_dl=dls.valid)

# Export keeping the Q/DQ nodes, then check what the file actually contains
path = export_qdq(quantized_model, sample, "model_qdq.onnx")
print(qdq_stats(path).as_dict())
# for a ResNet-18: {'n_quantize': 33, 'n_dequantize': 54, 'n_per_channel': 21,
#                   'n_nonzero_zero_point': 0, 'n_unquantized_conv_add': 0}
# (ResNet-18, torch 2.9.1, opset 18 — the Q/DQ counts track the exporter version; only n_per_channel,
#  n_nonzero_zero_point and n_unquantized_conv_add are pinned by tests)

# fraction of inputs where ONNX and PyTorch predict the same class
print(verify_qdq(quantized_model, path, samples, n_batches=8))

Naming the Precision

# One scale per weight tensor instead of one per output channel
quantizer = Quantizer(backend='pt2e', qscheme='per_tensor')

# Keep the classifier in floating point, quantize everything else to INT8
quantizer = Quantizer(backend='x86', weight_bits={'fc': 16})

# INT8 weights only, one scale per group of 64 weights (torchao, Linear layers)
quantizer = Quantizer(backend='torchao', weight_bits=8, act_bits=16, qscheme='per_group', group_size=64)

# What was actually applied — also attached to every model this quantizer produces
print(quantizer.spec.as_dict())

Where the Q/DQ Pairs Sit

qdq_placement is a different axis from the widths: it does not change how many bits a tensor keeps, it changes which edges of the graph carry a quantize/dequantize pair. Only the pt2e backend has it, because it is the only flow fasterai annotates itself.

Value What the graph gets
'per_op' (default) Every operator the flow annotates has its result quantized — what pt2e has always produced
'skip_conv_add' The residual branch — the addend produced by a single-user convolution partition — reaches its addition with no pair, at accumulator precision 
quantizer = Quantizer(backend='pt2e', qdq_placement='skip_conv_add', verbose=True)
model_q = quantizer.quantize(model, calibration_dl=dls.valid)
# pt2e: qdq_placement='skip_conv_add' left 8 of 8 addition(s) reading a convolution at accumulator precision

# ...and the file says the same thing, counted off the artifact rather than off the request
print(qdq_stats(export_qdq(model_q, sample, "resnet18_skip.onnx")).n_unquantized_conv_add)   # 8

Read it for exactly what it is:

  • It is opt-in, and it changes the arithmetic. The residual result is no longer rounded to INT8 before the addition. The effect on a trained model’s accuracy is unmeasured here.
  • What a runtime does with that graph is the runtime’s property. fasterai writes the edges; which operators an inference engine then fuses, and whether that costs or saves anything, is decided by that engine.
  • Post-training, nothing else moves. An observer records a range, it does not round, so every pair the graph keeps carries the same scale it would have had (pinned by a test). Under QAT that is not true: the fake-quantize modules alter the tensors, so removing one changes the statistics downstream of it.
  • It refuses rather than doing nothing. A model with no residual addition fed by a single-user convolution raises, so a recorded placement is always one that ran. The names it does not match today (concatenations, pooling and multiply edges, Linear-fed residuals) are future values of this argument, not new arguments.

QAT trains through the placement: pass the quantizer to QuantizeCallback with Quantizer(backend='pt2e', method='qat', qdq_placement='skip_conv_add').

Activations Left in Floating Point (pt2e)

skip_activations is the other axis that moves pairs rather than bits: it takes the Q/DQ pairs off the activations one named module produces.

quantizer = Quantizer(backend='pt2e', skip_activations=['features.0'], verbose=True)
model_q = quantizer.quantize(model, calibration_dl=dls.valid)
# pt2e: skip_activations='features.0' left 2 activation site(s) in floating point
#       (2 annotation entry(ies) cleared, 0 shared spec(s) swept)

A name selects the nodes torch attributes to that module or to anything under it; for each of them the output qspec, every consumer’s entry for it and every spec defined against a cleared edge are removed. The observer of a fused conv→BN→ReLU partition sits on one node of the partition, so name the block, not the convolution: a name nothing observes is refused, and the refusal says which name to use instead. The pass is idempotent, so it composes with qdq_placement='skip_conv_add' in either order — naming a residual branch and clearing the conv→add edge are the same two edges.

Read it for exactly what it is:

  • ONNX Runtime and TensorRT fuse a Q/DQ pair into an INT8 operator; with the pair gone, the convolution reading that tensor is no longer one of them. There is no post-condition on the file for it the way qdq_placement has one — the check runs on the prepared graph (no observer under the name, or a refusal), and the file shows it as one quantize and one dequantize node fewer per cleared tensor.
  • Torch’s own module knob does not express it. XNNPACKQuantizer.set_module_name(name, None) raises AssertionError: quantization_config == None is not supported yet, and even without that assertion it would clear the producer’s output qspec alone, leaving the consumer’s entry to quantize the same tensor one node later.
  • What it was measured on. torchvision mobilenet_v3_small (torch 2.9.1, torchvision 0.24.1, fasterai e10be99), one calibration draw of 128 images, 500 class-balanced validation images, the converted graph run on CPU: top-1 goes from 0.8% (4/500) to 65.8% (329/500) under the symmetric default, and from 6.0% (30/500) to 74.2% (371/500) with symmetric=False, when the stem block’s activations stay in floating point (skip_activations=['features.0'], 4 of 275 Q/DQ nodes removed). Per-channel INT8 on that same site scores the same, which is what identifies the per-tensor range of the site — 16 channels spanning 0.14 to 90.1 — as the cost rather than the bit width. One model; a second calibration draw moves the same way (1.6% to 68.2% symmetric, 3.2% to 73.2% affine).
  • The data-free alternative is equalisation. Cross-layer equalisation (Nagel et al., 2019) rescales neighbouring layers to even out such a spread instead of excluding the site; fasterai does not implement it.

Per-Layer Widths

A {layer_name: width} dict quantizes some layers and leaves the others in floating point. 16 is the only spelling for “leave this one alone”; the layers the dict does not name keep the uniform width, so the dict is an override list, not a whitelist.

# INT8 everywhere except the first feed-forward layer of each block (torchao, weight-only)
quantizer = Quantizer(backend='torchao', act_bits=16, verbose=True,
                      weight_bits={'layers.0.linear1': 16, 'layers.1.linear1': 16})
model_q = quantizer.quantize(model)
# torchao: applying int8_weight_only to 2/4 Linear layers (2 left in floating point:
#          layers.0.linear1, layers.1.linear1)
# torchao: quantized 2 layers

# ...and the same grammar on a CNN, where the legacy backends quantize convolutions too
Quantizer(backend='x86', weight_bits={'fc': 16}).quantize(model, calibration_dl=dls.valid)

# On the legacy backends a container name covers everything under it, in one entry
Quantizer(backend='x86', weight_bits={'layer4': 16}).quantize(model, calibration_dl=dls.valid)

Which layers a dict may name depends on the backend. torchao rewrites nn.Linear only (and never MultiheadAttention.out_proj, which it skips), so it takes those names and no others. The legacy backends take the module types their default mapping rewrites — the convolutions, Linear, and the normalizations and activations that mapping covers — plus any module containing one, because the FX flow resolves a module-name configuration by walking parents.

Naming anything else raises rather than being ignored, and so do the two other ways a dict can describe a model that was never quantized: asking for 8 inside a container already left at 16 (the outer 16 wins, so the 8 could not be honored), and a dict that leaves every layer the backend rewrites in floating point. method='dynamic' refuses a dict outright — quantize_dynamic reads no per-module configuration, so there is nothing for the dict to reach.

A precision the backend cannot honor raises when the Quantizer is built, naming one that can — see Precision.

What quantize Refuses

quantize either returns a quantized model or raises. It never hands back the model it was given, and it never attaches a precision to a model that does not carry it.

Situation What happens
The backend fails (an untraceable model, a model that cannot be copied) RuntimeError, chained to the cause the backend raised
The conversion produced no integer weight ValueError — every layer came back in floating point
method='dynamic' and no Linear/LSTM/GRU of the model came back integer-weighted ValueError naming what that flow rewrites
device= is anything but CPU ValueError — this flow calibrates and converts on the CPU
More than one convolution came back in floating point ValueError naming a backend that quantizes convolutions 
Quantizer(method='dynamic').quantize(conv_net)
# ValueError: method='dynamic' rewrites Linear/LSTM/GRU only, and this model has none: `quantize`
# would hand back a float model carrying a quantized model's provenance. Use method='static'.

Quantizer(backend='x86').quantize(model, calibration_dl=dls.valid, device='cuda')
# ValueError: `device='cuda'`: PyTorch quantization runs on CPU, and calibrating elsewhere leaves
# the converted model split across devices. Use device='cpu'.

One recovery is deliberate: when a backend refuses per-channel weights, quantize retries the whole flow per-tensor and records qscheme='per_tensor' on the spec, so the provenance describes the axis that actually ran.

See Also

Tests live in nbs/tests/test_quantizer.ipynb.