Deployable INT8 Export

Quantize a model to symmetric INT8 and export it as a QDQ ONNX graph a runtime can read

Overview

A quantized PyTorch model is not, by itself, deployable: the runtime that will actually serve it needs a file that describes the quantization. This notebook walks the whole path for the one precision cell that survives an ONNX export intact — W8A8, symmetric, per-channel weights — and checks, at each step, what was really produced rather than what was asked for:

1. Name the precision — Quantizer(backend='pt2e', weight_bits=8, act_bits=8, qscheme='per_channel', symmetric=True)
2. Export — export_qdq, which keeps the QuantizeLinear / DequantizeLinear pairs in the graph
3. Inspect — qdq_stats, which counts what is in the file
4. Verify — verify_qdq, which compares the exported graph’s predictions against PyTorch’s

Why symmetric: a symmetric quantizer gives every tensor a zero-point of 0, and some runtimes only accept quantized tensors whose zero-point is 0. Why per-channel: one scale per output channel costs a handful of extra numbers and gives each output channel its own dynamic range; it is the pt2e default. This notebook does not measure the accuracy difference between the two axes.

Note

The calibration data below is random noise, which keeps this notebook self-contained. Calibration decides the activation ranges of the quantized model, so a real run must calibrate on real data from your task — otherwise the scales describe noise, not your inputs. Everything else here (the graph, the node counts, the PyTorch/ONNX agreement) is unaffected by that choice.

import torch
import torch.nn as nn
from torchvision.models import resnet18

from fasterai.core.all import quant_spec
from fasterai.quantize.quantizer import Quantizer
from fasterai.export.all import export_onnx, export_qdq, qdq_stats, verify_qdq

The model

An untrained ResNet-18 — the numbers below are about the graph and its size, not about accuracy. torch.export captures a single batch size, so the calibration batches and the deployment sample all use the same one.

torch.manual_seed(0)
model = resnet18(weights=None).eval()

# `torch.export` captures ONE batch size: calibrate and deploy with that same one.
BATCH = 8
calibration_dl = [torch.randn(BATCH, 3, 224, 224) for _ in range(4)]
sample = torch.randn(BATCH, 3, 224, 224)

print(sum(p.numel() for p in model.parameters()), "parameters")

11689512 parameters

The FP32 baseline

Export the float model first, to have something to compare the quantized file against.

float_path = export_onnx(model, sample, "resnet18_fp32.onnx", dynamic_batch=False)
print(float_path, f"{float_path.stat().st_size / 1e6:.1f} MB")

resnet18_fp32.onnx 46.7 MB

1. Name the precision

The keyword-only arguments name the precision cell; backend names who applies it. Every argument here is the pt2e default, spelled out — asking for the default explicitly produces exactly the same model, and asking for something the backend cannot do raises immediately (see the last section).

quantizer.spec is the resolved request: one frozen record of what will be applied.

quantizer = Quantizer(backend='pt2e', weight_bits=8, act_bits=8,
                     qscheme='per_channel', symmetric=True)
qmodel = quantizer.quantize(model, calibration_dl=calibration_dl, max_calibration_samples=32)

print(quantizer.spec.as_dict())

{'backend': 'pt2e', 'method': 'static', 'weight_bits': 8, 'act_bits': 8, 'qscheme': 'per_channel', 'symmetric': True, 'group_size': None, 'layer_bits': None, 'qdq_placement': 'per_op', 'skip_activations': None, 'engine': None}

The same record is attached to the model that comes out, so the precision travels with the artifact rather than living in the notebook that produced it:

spec = quant_spec(qmodel)
print(spec.label, "| exportable:", spec.exports)

W8A8 | exportable: True

2. Export

export_qdq writes the quantize/dequantize pairs into the ONNX graph instead of folding them away. (export_onnx is the float path: its graph optimizer would fuse exactly the nodes we need to keep.)

path = export_qdq(qmodel, sample, "resnet18_qdq.onnx")
print(path, f"{path.stat().st_size / 1e6:.1f} MB",
      f"({float_path.stat().st_size / path.stat().st_size:.1f}x smaller than FP32)")

[torch.onnx] Obtain model graph for `GraphModule([...]` with `torch.export.export(..., strict=False)`...
[torch.onnx] Obtain model graph for `GraphModule([...]` with `torch.export.export(..., strict=False)`... ✅
[torch.onnx] Run decomposition...
[torch.onnx] Run decomposition... ✅
[torch.onnx] Translate the graph into ONNX...
[torch.onnx] Translate the graph into ONNX... ✅
Applied 20 of general pattern rewrite rules.
resnet18_qdq.onnx 11.9 MB (3.9x smaller than FP32)

3. Inspect what was produced

qdq_stats reads the file back and counts what the exporter actually wrote — the point being to check the artifact, not the intention:

stats = qdq_stats(path)
print(stats.as_dict())

{'n_quantize': 33, 'n_dequantize': 54, 'n_per_channel': 21, 'n_nonzero_zero_point': 0, 'n_unquantized_conv_add': 0}

• n_quantize / n_dequantize: the Q/DQ pairs a runtime reads to build its INT8 kernels.
• n_per_channel: 21 tensors carry one scale per channel — the 20 convolutions plus the classifier.
• n_nonzero_zero_point: 0 is the number that matters here. It is what symmetric=True promised, measured on the file rather than trusted.
• n_unquantized_conv_add: 0 — every Add in this graph reads its inputs through a Q/DQ pair, which is what the default placement (qdq_placement='per_op') produces. The last section moves that.

4. Verify

Node counts say the file is shaped right; they do not say it computes the same thing. verify_qdq runs both the PyTorch model and the exported graph and reports how often they predict the same class.

import warnings

probe = torch.randn(32, 3, 224, 224)
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter('always')
    agreement = verify_qdq(qmodel, path, probe, n_batches=4)

print(f"argmax agreement with PyTorch: {agreement:.3f} ({int(agreement * len(probe))}/{len(probe)} inputs)")
for w in caught: print(f"warning: {w.message}")

argmax agreement with PyTorch: 1.000 (32/32 inputs)
warning: The reference model predicts a single class for every input, so this agreement is 1.0 whatever the ONNX graph computes: the check is vacuous here. Use a trained model and inputs whose predictions vary, or compare logits directly.

32/32 inputs agree — as a proportion, 1.000, with a Wilson 95% interval of [0.89, 1.00]. Read it for exactly what it is:

• Here it is a smoke test, not a parity guard. This untrained ResNet-18 answers the same class to every noise input, so agreement is 1.000 whether or not the export is faithful — which is why verify_qdq warns above. What it does prove: the file loads, runs at the captured batch size, and produces the expected shape.
• It becomes a real parity guard on a trained model whose predictions vary across the probe set; a broken export then shows up as disagreement. To check faithfulness without a trained model, compare logits instead of argmax (that is what fasterai’s own test suite pins, with a tolerance calibrated on the measured residual).
• It is agreement with the QUANTIZED PyTorch model, not task accuracy. The accuracy cost of quantization is a separate measurement, against the float model, on real data.

Optional: where the Q/DQ pairs sit

Every argument so far said how many bits. qdq_placement says which edges of the graph carry a Q/DQ pair. Its default, 'per_op', is what every export above produced: the result of each operator the flow annotates is quantized. 'skip_conv_add' leaves one edge alone — the residual branch, the addend produced by a single-user convolution partition — so that result reaches its addition at accumulator precision.

It is opt-in, and it changes the arithmetic. What follows measures what it does to the graph.

skip = Quantizer(backend='pt2e', qdq_placement='skip_conv_add', verbose=True).quantize(
    model, calibration_dl=calibration_dl, max_calibration_samples=32)
skip_path = export_qdq(skip, sample, "resnet18_qdq_skip.onnx")

print(quant_spec(skip).qdq_placement)
print(stats.as_dict())
print(qdq_stats(skip_path).as_dict())

pt2e: capturing the model with torch.export
pt2e: qdq_placement='skip_conv_add' left 8 of 8 addition(s) reading a convolution at accumulator precision
pt2e: calibrating with up to 32 samples
pt2e: converting to a quantized graph
[torch.onnx] Obtain model graph for `GraphModule([...]` with `torch.export.export(..., strict=False)`...
[torch.onnx] Obtain model graph for `GraphModule([...]` with `torch.export.export(..., strict=False)`... ✅
[torch.onnx] Run decomposition...
[torch.onnx] Run decomposition... ✅
[torch.onnx] Translate the graph into ONNX...
[torch.onnx] Translate the graph into ONNX... ✅
Applied 20 of general pattern rewrite rules.
skip_conv_add
{'n_quantize': 33, 'n_dequantize': 54, 'n_per_channel': 21, 'n_nonzero_zero_point': 0, 'n_unquantized_conv_add': 0}
{'n_quantize': 25, 'n_dequantize': 46, 'n_per_channel': 21, 'n_nonzero_zero_point': 0, 'n_unquantized_conv_add': 8}

ResNet-18 has eight residual additions, and this placement clears all eight: n_unquantized_conv_add goes from 0 to 8, and the file loses exactly eight QuantizeLinear and eight DequantizeLinear nodes. n_per_channel and n_nonzero_zero_point do not move — the weights are quantized exactly as before, and every zero-point is still 0.

Three things to keep straight before using it:

• The accuracy cost on a trained model is unmeasured here. This notebook quantizes an untrained ResNet-18 on random calibration data; it can show you the graph, not the accuracy.
• What a runtime does with that graph is the runtime’s property. fasterai decides where the pairs are written. Which operators an inference engine then fuses is that engine’s decision, and whether it costs or saves anything has to be measured on your target.
• It refuses rather than doing nothing. A model with no residual addition fed by a single-user convolution raises, so a recorded placement is one that ran — and export_qdq re-checks the produced file against the spec, keeping no file if the two disagree.

When a precision cannot be honored

The grammar is validated against a support matrix (see Precision), so a request a backend cannot apply fails at construction — with the name of a backend that can — instead of being quietly replaced by a nearby precision:

try:
    Quantizer(backend='x86', symmetric=True)
except ValueError as e:
    print(e)

backend='x86' W8A8 cannot honor symmetric=True — its observers are affine by construction: activations carry a non-zero zero-point. The backend(s) that can: ['pt2e', 'torchao'].

The same applies to the export. Not every precision has a QDQ form: INT4 weights have no Q/DQ pair in opset 18, and a weight-only model has no activation scales to write. export_qdq reads the spec attached to the model and refuses, rather than writing a graph that describes something else:

weight_only = Quantizer(backend='torchao', method='int8_weight_only').quantize(
    nn.Sequential(nn.Linear(64, 10)).eval())
try:
    export_qdq(weight_only, torch.randn(1, 64), "weight_only.onnx")
except ValueError as e:
    print(e)

export_qdq cannot write this model: it was quantized W8A16 with backend='torchao'. INT8 weight-only: the activations stay in floating point, so there is no activation Q/DQ pair to export. This is the one torchao cell that honors a per-layer `weight_bits` dict, over the Linear layers torchao rewrites. Quantize with Quantizer(backend='pt2e') for a graph this export can write.

Summary

Step	Tool	What it gives you
Name the precision	Quantizer(backend='pt2e', weight_bits=8, act_bits=8, qscheme='per_channel', symmetric=True)	A validated QuantSpec, refused up front if the backend cannot honor it
Quantize	Quantizer.quantize(model, calibration_dl)	A quantized graph, tagged with the precision that produced it
Export	export_qdq(model, sample, path)	An ONNX file whose Q/DQ pairs survived the export
Inspect	qdq_stats(path)	Node counts, per-channel count, zero-point audit
Verify	verify_qdq(model, path, samples)	Argmax agreement between the exported graph and PyTorch
Move the pairs	Quantizer(..., qdq_placement='skip_conv_add')	The residual conv→add edge left unquantized, checked on the produced file

From a single run of an untrained ResNet-18 with random calibration and random probe inputs, on torch 2.9.1 / onnxruntime CPU:

• baseline 46.7 MB — the FP32 ONNX graph from export_onnx (opset 17, static batch)
• 11.9 MB — the QDQ INT8 graph, 3.9x smaller ON DISK; this run measures no latency and no accuracy
• 0 non-zero zero-points in the exported file (what symmetric=True promised)
• 1.000 argmax agreement (32/32 inputs) with the quantized PyTorch model — vacuous on this untrained model, see the note above
• 0 → 8 unquantized conv→add edges with qdq_placement='skip_conv_add', a graph-shape count read off the file; this run measures no accuracy and no latency for that option either

Two things to carry over to a real model:

• Calibrate on real data. The scales come from the calibration batches, and nothing downstream can repair scales fitted to the wrong distribution.
• Mind the batch size. torch.export specialises the graph to the batch size it was captured with, so calibrate, export and serve with the same one.

---

See Also

• Precision - The support matrix, and what each backend can honor
• Quantizer - Every backend and method, including the legacy FX flow
• ONNX Exporter - export_onnx, export_qdq, qdq_stats, verify_qdq
• Quantization Methods Compared - Size and latency of several quantized configurations on one box