import math
import torch, torch.nn as nn
from fastai.vision.all import *
from fasterai.quantize.fake_quantizer import FakeQuantizer
from fasterai.core.precision import fake_quant_specFakeQuantizer
What 8-, 4- and 2-bit rounding costs three fine-tuned classifiers on Imagenette, measured through fastai
FakeQuantizer rounds weights and activations onto a narrower grid and leaves the result in floating point. The model keeps its float32 tensors and its ordinary modules; while it is rounded it also holds a floating-point copy of every weight it touched, so it takes more memory than it did before. When act_bits is set it additionally hooks every rounded module and rounds its output on each forward pass; with act_bits=None no hook is registered. The one thing it measures is what a width costs in accuracy.
This page rounds three fine-tuned classifiers — vgg16_bn, resnet18 and efficientnet_b0 — to the same grid of widths and reports what each one costs on Imagenette.
Scope, before any number. Each architecture is fine-tuned once, each configuration is validated once, and every static row observes one fixed calibration set. Three architectures trained once each is an observation about these three fine-tuned models, not a controlled comparison between architectures: nothing here separates what the architecture does from what this particular fine-tune did. The page reports accuracy and prediction changes, and measures no latency.
Setup
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'torch {torch.__version__}')
print(f'device {device} ({torch.cuda.get_device_name(0) if device.type == "cuda" else "cpu"})')torch 2.9.1+cu128
device cuda (NVIDIA GeForce RTX 5090)Imagenette at 160 pixels, one fine_tune(1) per architecture — one frozen epoch and one unfrozen one, enough to give each model a floating-point accuracy to be read against.
Calibration gets its own fixed loader rather than a handful of shuffled training batches. dls.test_dl applies validation-style preprocessing, so the five batches are the same on every call and every static row below observes exactly the same activations. What the calibration set decides measures how much that choice is worth.
path = untar_data(URLs.IMAGENETTE_160)
dls = ImageDataLoaders.from_folder(path, valid='val', item_tfms=Resize(160), bs=32)
N = len(dls.valid_ds)
# one fixed calibration set, so every static row on this page observes the same batches
files = sorted(get_image_files(path/'train'))
calib_dl = dls.test_dl(files[::len(files)//160][:160], bs=32)
print(f'{N} validation images, {len(dls.train_ds)} training images')
print(f'calibration set: {len(calib_dl)} batches, {len({f.parent.name for f in files[::len(files)//160][:160]})} classes')3925 validation images, 9469 training images
calibration set: 5 batches, 10 classesdef wilson(k, n, z=1.96):
"Wilson 95% interval for k of n, as percentages"
p, d = k / n, 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z / d * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return 100 * max(0., c - h), 100 * min(1., c + h)
def finetune(arch):
"A fastai `vision_learner` fine-tuned for one cycle on Imagenette"
set_seed(42, reproducible=True)
learn = vision_learner(dls, arch, metrics=accuracy)
with learn.no_bar(), learn.no_logging():
learn.fine_tune(1)
return learn
def score(learn):
"Validation accuracy of `learn`, as a percentage and as k/n"
with learn.no_bar():
acc = float(learn.validate()[1])
return 100 * acc, round(acc * N), N
def predictions(learn):
"Predicted class for every validation image"
with learn.no_bar():
preds, _ = learn.get_preds(dl=dls.valid)
return preds.argmax(1)
learners = {'vgg16_bn': finetune(vgg16_bn),
'resnet18': finetune(resnet18),
'efficientnet_b0': finetune(efficientnet_b0)}
float_pred = {name: predictions(learn) for name, learn in learners.items()}
for name, learn in learners.items():
pct, k, n = score(learn)
lo, hi = wilson(k, n)
print(f'{name:<18}{pct:.2f}% ({k}/{n}) Wilson 95% [{lo:.2f}, {hi:.2f}]')vgg16_bn 96.92% (3804/3925) Wilson 95% [96.33, 97.41]
resnet18 95.95% (3766/3925) Wilson 95% [95.29, 96.52]
efficientnet_b0 95.44% (3746/3925) Wilson 95% [94.74, 96.05]Those three are the references every row below is read against. The Wilson 95% intervals span about a point each, which is the yardstick for reading the small differences further down: at this n, a handful of images is well inside the interval.
What rounding does not do
The model does not get smaller. quantize_model() writes rounded values back into the same float32 tensors and registers a buffer holding the original of every weight it rounds, so the model grows while it is rounded and shrinks back on remove().
def nbytes(m):
"Bytes held by every parameter and buffer of `m`"
return (sum(p.numel() * p.element_size() for p in m.parameters())
+ sum(b.numel() * b.element_size() for b in m.buffers()))
model = learners['resnet18'].model
before = {k: v.detach().clone() for k, v in model.state_dict().items()}
fq = FakeQuantizer(model, weight_bits=8, act_bits=8, qscheme='per_channel')
fq.calibrate(calib_dl, n_batches=5)
fq.quantize_model()
print(f'dtype while rounded {next(model.parameters()).dtype}')
print(f'bytes while rounded {nbytes(model):,}')
print(f'spec while rounded {fake_quant_spec(model).label}')
fq.remove()
print(f'bytes after remove() {nbytes(model):,}')
print(f'spec after remove() {fake_quant_spec(model)}')
print(f'weights restored {all(torch.equal(before[k], v) for k, v in model.state_dict().items())}')dtype while rounded torch.float32
bytes while rounded 93,672,288
spec while rounded W8A8
bytes after remove() 46,886,832
spec after remove() None
weights restored TrueRounded, this ResNet-18 holds 93,672,288 bytes of parameters and buffers against 46,886,832 before. remove() puts the original weights back byte for byte and drops the spec.
Four more things stay outside the rounding:
- Biases stay in floating point. An integer kernel would carry them at int32; this class does not model that.
- BatchNorm is not folded into the convolution it follows. Folding changes the weights that get rounded, so a folded model and an unfolded one give different answers at the same width — [
BN_Folder](https://FasterAI-Labs.github.io/fasterai/misc/bn_folding.html#bn_folder) does the folding. nn.Embeddingandnn.ConvTranspose2dare outside the defaultlayer_type, which is(nn.Conv2d, nn.Linear). A model holding them keeps those tensors byte-identical while the report still names a width.- A rounded model is not a QAT model. Its weights are baked: they sit on the grid now, and training would walk them straight off it. Quantization-aware training is not in this version.
print_precision() names the width every layer carries. Here layer_bits holds the first convolution in floating point:
fq = FakeQuantizer(model, weight_bits=8, act_bits=8, qscheme='per_channel',
layer_bits={'0.0': None})
fq.calibrate(calib_dl, n_batches=5)
fq.quantize_model()
fq.print_precision()
fq.remove()
Simulated Precision Report:
--------------------------------------------------------------------------------
Layer Type Weight Act Weight axis
--------------------------------------------------------------------------------
0.0 Conv2d float 8 bits -
0.4.0.conv1 Conv2d 8 bits 8 bits per_channel
0.4.0.conv2 Conv2d 8 bits 8 bits per_channel
0.4.1.conv1 Conv2d 8 bits 8 bits per_channel
0.4.1.conv2 Conv2d 8 bits 8 bits per_channel
0.5.0.conv1 Conv2d 8 bits 8 bits per_channel
0.5.0.conv2 Conv2d 8 bits 8 bits per_channel
0.5.0.downsample.0 Conv2d 8 bits 8 bits per_channel
0.5.1.conv1 Conv2d 8 bits 8 bits per_channel
0.5.1.conv2 Conv2d 8 bits 8 bits per_channel
0.6.0.conv1 Conv2d 8 bits 8 bits per_channel
0.6.0.conv2 Conv2d 8 bits 8 bits per_channel
0.6.0.downsample.0 Conv2d 8 bits 8 bits per_channel
0.6.1.conv1 Conv2d 8 bits 8 bits per_channel
0.6.1.conv2 Conv2d 8 bits 8 bits per_channel
0.7.0.conv1 Conv2d 8 bits 8 bits per_channel
0.7.0.conv2 Conv2d 8 bits 8 bits per_channel
0.7.0.downsample.0 Conv2d 8 bits 8 bits per_channel
0.7.1.conv1 Conv2d 8 bits 8 bits per_channel
0.7.1.conv2 Conv2d 8 bits 8 bits per_channel
1.4 Linear 8 bits 8 bits per_channel
1.8 Linear 8 bits 8 bits per_channel
--------------------------------------------------------------------------------
Overall W8A8
Rounded in floating point: every tensor is still stored at its original width, and the model holds a floating-point copy of every weight it rounds until remove().The widths, three models
Each configuration is built on the fine-tuned model, validated, and removed before the next one starts. Alongside the accuracy, each row counts the validation images whose predicted class differs from the floating-point model’s — a row that changed nothing would show zero, so the count is what separates a real effect from a no-op.
def measure(learn, ref=None, calib=None, **kw):
"Round `learn.model` to one configuration, validate it, and restore the floating-point weights"
fq = FakeQuantizer(learn.model, **kw)
try:
if fq.observer == 'static' and fq.act_bits is not None:
fq.calibrate(calib if calib is not None else calib_dl, n_batches=5)
fq.quantize_model()
pct, k, n = score(learn)
flips = None if ref is None else int((predictions(learn) != ref).sum())
return pct, k, n, flips
finally:
fq.remove()
CONFIGS = {
'W8A8 static, per_channel': dict(weight_bits=8, act_bits=8, qscheme='per_channel'),
'W8 weights-only, per_channel': dict(weight_bits=8, act_bits=None, qscheme='per_channel'),
'W8 weights-only, per_tensor': dict(weight_bits=8, act_bits=None, qscheme='per_tensor'),
'W4A8 static, per_channel': dict(weight_bits=4, act_bits=8, qscheme='per_channel'),
'W4 weights-only, per_channel': dict(weight_bits=4, act_bits=None, qscheme='per_channel'),
'W4 weights-only, per_tensor': dict(weight_bits=4, act_bits=None, qscheme='per_tensor'),
'W2A8 static, per_channel': dict(weight_bits=2, act_bits=8, qscheme='per_channel'),
'W2 weights-only, per_channel': dict(weight_bits=2, act_bits=None, qscheme='per_channel'),
}
rows = {'floating point': {n: (*score(l), 0) for n, l in learners.items()}}
for label, kw in CONFIGS.items():
rows[label] = {n: measure(l, ref=float_pred[n], **kw) for n, l in learners.items()}
print(f'{"":<32}' + ''.join(f'{name:<19}' for name in learners))
for label, per_arch in rows.items():
cells = ''.join(f'{pct:.2f}% {k}/{n}'.ljust(19) for pct, k, n, _ in
(per_arch[name] for name in learners))
print(f'{label:<32}{cells}')
print(f'\nimages whose predicted class differs from the floating-point model, of {N}')
print(f'{"":<32}' + ''.join(f'{name:<19}' for name in learners))
for label, per_arch in rows.items():
print(f'{label:<32}' + ''.join(f'{per_arch[name][3]}'.ljust(19) for name in learners)) vgg16_bn resnet18 efficientnet_b0
floating point 96.92% 3804/3925 95.95% 3766/3925 95.44% 3746/3925
W8A8 static, per_channel 96.97% 3806/3925 95.34% 3742/3925 64.36% 2526/3925
W8 weights-only, per_channel 97.04% 3809/3925 95.90% 3764/3925 95.39% 3744/3925
W8 weights-only, per_tensor 96.97% 3806/3925 96.00% 3768/3925 95.52% 3749/3925
W4A8 static, per_channel 94.09% 3693/3925 88.92% 3490/3925 30.96% 1215/3925
W4 weights-only, per_channel 94.39% 3705/3925 89.96% 3531/3925 45.35% 1780/3925
W4 weights-only, per_tensor 22.78% 894/3925 26.50% 1040/3925 10.78% 423/3925
W2A8 static, per_channel 10.39% 408/3925 10.06% 395/3925 11.54% 453/3925
W2 weights-only, per_channel 10.42% 409/3925 10.06% 395/3925 10.65% 418/3925
images whose predicted class differs from the floating-point model, of 3925
vgg16_bn resnet18 efficientnet_b0
floating point 0 0 0
W8A8 static, per_channel 29 52 1386
W8 weights-only, per_channel 6 9 50
W8 weights-only, per_tensor 16 16 67
W4A8 static, per_channel 182 379 2727
W4 weights-only, per_channel 171 331 2140
W4 weights-only, per_tensor 3027 2876 3509
W2A8 static, per_channel 3522 3536 3474
W2 weights-only, per_channel 3521 3536 3498 At 8 bits on the weights alone, every row lands within a handful of images of its reference, in both directions: vgg16_bn goes from 3804/3925 to 3809/3925 correct, resnet18 from 3766/3925 to 3764/3925, efficientnet_b0 from 3746/3925 to 3744/3925. All six differences sit inside the Wilson intervals printed above. The flip counts show this is not a no-op — 6, 9 and 50 images changed class — so what the accuracy column shows is many small changes very nearly cancelling.
Adding 8-bit activations splits the three. vgg16_bn and resnet18 stay put at 96.97% (3806/3925) and 95.34% (3742/3925), while efficientnet_b0 drops to 64.36% (2526/3925) with 1386 images changing class against 50 for its weights-only row. Its weights survive 8 bits; its activations do not. That contrast is a one-variable comparison — the same model, the same calibration set, act_bits alone moved — and it is by far the largest effect on this page.
At 4 bits the order across the three models is the one the 8-bit rows already had, and the gaps widen: 94.39% (3705/3925), 89.96% (3531/3925) and 45.35% (1780/3925) per channel. resnet18 gives up more than vgg16_bn here, and with one fine-tune each that is a fact about these two trained models rather than a ranking of the architectures.
At 2 bits none of the three separates the classes any more, and roughly nine validation images in ten have changed class.
Two guards
Seven apply-and-remove cycles have run against each model by now. Re-scoring the untouched models has to give the opening numbers back:
for name, learn in learners.items():
pct, k, n = score(learn)
print(f'{name:<18}{pct:.2f}% ({k}/{n})')vgg16_bn 96.92% (3804/3925)
resnet18 95.95% (3766/3925)
efficientnet_b0 95.44% (3746/3925)It does. Nothing leaked between configurations.
The second guard checks the other direction — that the rounding actually landed on a grid instead of being a near-no-op. Under per_channel each output row carries its own scale, so each row can take at most 2**bits distinct values:
def row_values(learn, qscheme, **kw):
"Largest number of distinct values any single scaled row of a weight takes"
fq = FakeQuantizer(learn.model, qscheme=qscheme, **kw) if kw else None
try:
if fq is not None: fq.quantize_model()
worst = 0
for m in learn.model.modules():
if not isinstance(m, (nn.Conv2d, nn.Linear)): continue
w = m.weight.detach()
rows_ = w.flatten(1) if qscheme == 'per_channel' else w.reshape(1, -1)
worst = max(worst, max(int(r.unique().numel()) for r in rows_))
return worst
finally:
if fq is not None: fq.remove()
for name, learn in learners.items():
out = [f'float {row_values(learn, "per_channel"):7d}']
for bits in (8, 4, 2):
got = row_values(learn, 'per_channel', weight_bits=bits, act_bits=None)
out.append(f'W{bits} {got:4d} <= {2 ** bits}')
print(f'{name:<18}' + ' '.join(out))vgg16_bn float 4608 W8 228 <= 256 W4 15 <= 16 W2 3 <= 4
resnet18 float 4608 W8 224 <= 256 W4 15 <= 16 W2 3 <= 4
efficientnet_b0 float 2560 W8 224 <= 256 W4 15 <= 16 W2 3 <= 4In floating point the widest row of these models takes 4608, 4608 and 2560 distinct values; rounded, no row exceeds its budget of 256, 16 and 4.
Where the scale comes from
qscheme picks the axis the scale is fitted on. per_tensor fits one scale to the whole weight tensor; per_channel fits one per output row.
At 8 bits the axis barely matters, and it does not order consistently. per_tensor finishes above per_channel on resnet18, 96.00% (3768/3925) against 95.90% (3764/3925), and on efficientnet_b0, 95.52% (3749/3925) against 95.39% (3744/3925) — and below it on vgg16_bn, 96.97% (3806/3925) against 97.04% (3809/3925). Every one of those differences is a few images inside a Wilson interval about a point wide, so on all three of these models the 8-bit rows do not establish that either axis dominates.
At 4 bits the axis decides the outcome. per_tensor falls to 22.78% (894/3925), 26.50% (1040/3925) and 10.78% (423/3925), against 94.39% (3705/3925), 89.96% (3531/3925) and 45.35% (1780/3925) per channel. One scale has to cover the loudest row of the tensor, and at four bits the quiet rows round away entirely:
def dead_filters(learn, **kw):
"Output filters whose weights are all zero"
fq = FakeQuantizer(learn.model, **kw) if kw else None
try:
if fq is not None: fq.quantize_model()
return sum(int((m.weight.flatten(1).abs().sum(1) == 0).sum())
for m in learn.model.modules() if isinstance(m, (nn.Conv2d, nn.Linear)))
finally:
if fq is not None: fq.remove()
def faint_filters(learn):
"Output filters whose largest floating-point weight is below 1e-6"
return sum(int((m.weight.flatten(1).abs().amax(1) < 1e-6).sum())
for m in learn.model.modules() if isinstance(m, (nn.Conv2d, nn.Linear)))
for name, learn in learners.items():
base = dead_filters(learn)
pt = dead_filters(learn, weight_bits=4, act_bits=None, qscheme='per_tensor')
pc = dead_filters(learn, weight_bits=4, act_bits=None, qscheme='per_channel')
print(f'{name:<18}float {base:4d} faint (<1e-6) {faint_filters(learn):4d} '
f'W4 per_tensor {pt:5d} W4 per_channel {pc:4d}')vgg16_bn float 0 faint (<1e-6) 11 W4 per_tensor 81 W4 per_channel 6
resnet18 float 0 faint (<1e-6) 6 W4 per_tensor 15 W4 per_channel 4
efficientnet_b0 float 0 faint (<1e-6) 0 W4 per_tensor 413 W4 per_channel 0No filter is empty in floating point on any of the three. per_channel empties 6 and 4 on the two models that already hold 11 and 6 rows whose largest weight is below 1e-6, and empties none on efficientnet_b0, which has no such row — those are rows the scale floor collapses, not rows the 4-bit grid destroyed. per_tensor empties 81, 15 and 413, and the accuracy rows above are what that costs.
Groups, and a refusal
qscheme='per_group' cuts each row into fixed blocks that share one scale, so the block size has to divide the row. A convolution row is in_channels times the kernel height times the kernel width, and a ResNet-18 stem gives an odd one. FakeQuantizer says so before it writes anything:
try:
FakeQuantizer(learners['resnet18'].model, weight_bits=4, act_bits=None,
qscheme='per_group', group_size=64)
except ValueError as e:
print(e)group_size=64 does not divide the 147 weights of a row of '0.0': pass a group_size that divides it, or qscheme='per_channel'.Narrowing layer_type to the layers whose rows do divide is one way through — here, the two nn.Linear layers of the fastai head:
for name, learn in learners.items():
total = sum(m.weight.numel() for m in learn.model.modules()
if isinstance(m, (nn.Conv2d, nn.Linear)))
head = sum(m.weight.numel() for m in learn.model.modules() if isinstance(m, nn.Linear))
pct, k, n, flips = measure(learn, ref=float_pred[name], weight_bits=4, act_bits=None,
qscheme='per_group', group_size=64, layer_type=nn.Linear)
print(f'{name:<18}{pct:.2f}% ({k}/{n}) {flips:3d} flips '
f'rounded {head:,}/{total:,} weights = {100 * head / total:.2f}%')vgg16_bn 96.97% (3806/3925) 11 flips rounded 529,408/15,239,872 weights = 3.47%
resnet18 95.77% (3759/3925) 13 flips rounded 529,408/11,696,320 weights = 4.53%
efficientnet_b0 95.36% (3743/3925) 17 flips rounded 1,315,840/5,272,032 weights = 24.96%The fraction rounded is worth reading before the accuracy. On the two convolutional stacks the head is 3.47% and 4.53% of the Conv2d and Linear weights, so those rows say little. On efficientnet_b0 it is 24.96% — a quarter of the weights taken to 4 bits in groups of 64, for 95.36% (3743/3925) against a 95.44% (3746/3925) reference and 17 images changing class.
One layer held at floating point
layer_bits maps a layer name to its own width, and None holds that layer in floating point. The first convolution is a common one to hold out, since every later layer reads through it — though it is the smallest tensor in only one of these three models:
for name, learn in learners.items():
named = [(n, m) for n, m in learn.model.named_modules()
if isinstance(m, (nn.Conv2d, nn.Linear))]
first = named[0]
smallest = min(named, key=lambda nm: nm[1].weight.numel())
pct, k, n, flips = measure(learn, ref=float_pred[name], weight_bits=4, act_bits=None,
qscheme='per_channel', layer_bits={first[0]: None})
print(f'{name:<18}{pct:.2f}% ({k}/{n}) first conv {first[0]!r} {first[1].weight.numel():,} weights '
f'smallest {smallest[0]!r} {smallest[1].weight.numel():,}')vgg16_bn 95.39% (3744/3925) first conv '0.0.0' 1,728 weights smallest '0.0.0' 1,728
resnet18 92.99% (3650/3925) first conv '0.0' 9,408 weights smallest '1.8' 5,120
efficientnet_b0 47.26% (1855/3925) first conv '0.0.0.0' 864 weights smallest '0.0.1.0.block.1.fc1' 256It is the smallest on vgg16_bn at 1,728 weights; on resnet18 its 9,408 weights are more than the 5,120 of the head’s last layer, and on efficientnet_b0 its 864 are more than the 256 of a squeeze-and-excitation projection.
Holding it out of the 4-bit rounding moves vgg16_bn from 94.39% (3705/3925) to 95.39% (3744/3925), resnet18 from 89.96% (3531/3925) to 92.99% (3650/3925), and efficientnet_b0 from 45.35% (1780/3925) to 47.26% (1855/3925). It buys back part of what 4 bits cost on the two convolutional stacks, and leaves efficientnet_b0 far from its reference.
Where EfficientNet loses it
The W8A8 row said the activations are what efficientnet_b0 cannot carry at 8 bits. Hooking every rounded site over the fixed calibration set shows where their ranges are widest:
eff = learners['efficientnet_b0']
sites = {n: m for n, m in eff.model.named_modules() if isinstance(m, (nn.Conv2d, nn.Linear))}
def act_ranges(loader, n_batches=5):
"Smallest and largest activation each rounded site emits over `n_batches`"
seen = {}
def watch(name):
def hook(_, inp, out):
lo, hi = out.detach().amin().item(), out.detach().amax().item()
if name in seen: lo, hi = min(seen[name][0], lo), max(seen[name][1], hi)
seen[name] = (lo, hi)
return hook
handles = [m.register_forward_hook(watch(n)) for n, m in sites.items()]
eff.model.eval()
with torch.no_grad():
for i, batch in enumerate(loader):
if i >= n_batches: break
eff.model(batch[0])
for h in handles: h.remove()
return sorted(seen.items(), key=lambda kv: kv[1][1] - kv[1][0], reverse=True)
widest = act_ranges(calib_dl)
for name, (lo, hi) in widest[:5]:
m = sites[name]
kind = 'depthwise' if m.groups == m.in_channels > 1 else 'pointwise' if m.kernel_size == (1, 1) else 'conv'
print(f'{name:<22}[{lo:8.1f}, {hi:8.1f}] {kind:<10}k={m.kernel_size} groups={m.groups}')
print('\nfirst-ranked site on three shuffled five-batch draws from dls.train')
for draw in range(3):
print(f' draw {draw + 1}: ' + ', '.join(n for n, _ in act_ranges(dls.train)[:3]))0.0.4.0.block.0.0 [ -595.6, 163.5] pointwise k=(1, 1) groups=1
0.0.5.0.block.0.0 [ -110.9, 107.8] pointwise k=(1, 1) groups=1
0.0.3.0.block.0.0 [ -88.6, 88.4] pointwise k=(1, 1) groups=1
0.0.4.2.block.0.0 [ -103.1, 51.6] pointwise k=(1, 1) groups=1
0.0.2.0.block.0.0 [ -81.9, 69.6] pointwise k=(1, 1) groups=1
first-ranked site on three shuffled five-batch draws from dls.train
draw 1: 0.0.4.0.block.0.0, 0.0.2.0.block.0.0, 0.0.3.0.block.0.0
draw 2: 0.0.4.0.block.0.0, 0.0.2.0.block.0.0, 0.0.3.0.block.0.0
draw 3: 0.0.2.0.block.0.0, 0.0.4.0.block.0.0, 0.0.3.0.block.0.0On this calibration set the five widest are all 1x1 pointwise convolutions with groups=1, the expansion at the head of an inverted-residual block, and the widest runs from -595.6 to 163.5. One 8-bit grid stretched over that span leaves little resolution for the values near zero, which is most of them.
The ranking is a property of the data, not of the model alone: the three shuffled draws printed above do not agree on which site is widest. Read it as one observation from one calibration set.
layer_act_bits holds named sites at floating point the way layer_bits does for weights:
for k_sites in (0, 1, 3, 5, 10, 20):
pct, k, n, flips = measure(eff, ref=float_pred['efficientnet_b0'],
weight_bits=8, act_bits=8, qscheme='per_channel',
layer_act_bits={name: None for name, _ in widest[:k_sites]} or None)
print(f'{k_sites:2d} sites in floating point: {pct:.2f}% ({k}/{n}) {flips} flips') 0 sites in floating point: 64.36% (2526/3925) 1386 flips
1 sites in floating point: 63.82% (2505/3925) 1408 flips
3 sites in floating point: 64.48% (2531/3925) 1385 flips
5 sites in floating point: 64.15% (2518/3925) 1391 flips
10 sites in floating point: 80.08% (3143/3925) 773 flips
20 sites in floating point: 80.79% (3171/3925) 724 flipsThis does not behave the way the ranking suggests. Holding out the single widest site gives 2505/3925 correct against 2526/3925 for holding out none — slightly worse — and 3 and 5 sites give 2531/3925 and 2518/3925. All four sit inside the calibration spread measured in the next section, so the first five sites change nothing this page can resolve. The move appears at 10 sites, 80.08% (3143/3925), and barely grows at 20, 80.79% (3171/3925), still well short of the 95.44% (3746/3925) the model starts from.
So the widest-range ranking does not locate the damage: holding out the top of it is not what recovers the model, and twenty sites in floating point do not undo what 8-bit activations cost this architecture.
What the calibration set decides
Every static row above used one fixed calibration set. Repeating a single configuration on five shuffled, augmented draws from dls.train shows what that choice was worth:
draws = [measure(eff, weight_bits=8, act_bits=8, qscheme='per_channel', calib=dls.train)
for _ in range(5)]
for i, (pct, k, n, _) in enumerate(draws):
print(f'draw {i + 1}: {pct:.2f}% ({k}/{n})')
ks = [k for _, k, _, _ in draws]
fixed = rows['W8A8 static, per_channel']['efficientnet_b0']
print(f'spread over five shuffled draws: {max(ks) - min(ks)} images '
f'= {100 * (max(ks) - min(ks)) / N:.2f} points')
print(f'the fixed calibration set gives: {fixed[0]:.2f}% ({fixed[1]}/{fixed[2]})')draw 1: 48.31% (1896/3925)
draw 2: 49.43% (1940/3925)
draw 3: 50.93% (1999/3925)
draw 4: 50.27% (1973/3925)
draw 5: 48.97% (1922/3925)
spread over five shuffled draws: 103 images = 2.62 points
the fixed calibration set gives: 64.36% (2526/3925)Two things. The five shuffled draws spread over 103 images, 2.62 points, purely from which batches were drawn — that is the noise floor for every A8 number on this page, and no conclusion here rests on an A8 difference smaller than it. The weights-versus-activations contrast clears it by a wide margin, 3744/3925 against 2526/3925; the first four rows of the activation sweep above do not clear it at all.
Second, the fixed calibration set does not land inside that spread: 64.36% (2526/3925) against draws running from 48.31% (1896/3925) to 50.93% (1999/3925). The fixed set applies validation-style preprocessing while dls.train applies training augmentation, so the two observe different activation ranges. Which data the scales are frozen on moved this configuration further than any draw did.
Summary
| Call | What it does |
|---|---|
FakeQuantizer(model, weight_bits=8, act_bits=8) |
a quantizer bound to this model; None on either width leaves those tensors in floating point |
fq.calibrate(calib_dl, n_batches=5) |
observes activation ranges and freezes the static scales; takes a fastai DataLoaders or one of its loaders |
fq.quantize_model() |
rounds the weights in place and hooks the activations |
fq.remove() |
restores the floating-point weights and drops every hook and buffer |
fq.print_precision() |
the per-layer width report |
qscheme='per_tensor' / 'per_channel' / 'per_group' |
one scale for the tensor, one per output row, or one per block of a row |
group_size=64 |
the block size per_group shares a scale over; it has to divide the row |
layer_type=nn.Linear |
narrows the modules that carry the rounding; the default is (nn.Conv2d, nn.Linear) |
layer_bits={name: None} |
one weight width per layer, None holding it at floating point |
layer_act_bits={name: None} |
the same, for that layer’s activations |
observer='dynamic' |
recomputes activation scales per batch instead of freezing them; not exercised on this page |
What this page measured, and what it did not: accuracy and changed predictions, on three models, one fine-tune each. FakeQuantizer does not make the model smaller; when act_bits is set it adds a rounding op to every hooked activation, and this page does not measure latency.
Measured on: Imagenette-160, 9469 training images and 3925 validation images, at 160 pixels with batch size 32, on the torch build and device printed at the top of the page. One
fine_tune(1)per architecture underset_seed(42, reproducible=True), then onelearn.validate()per configuration on the same 3925 images. Static rows share one fixed 5-batch calibration set built withdls.test_dl; the measured draw-to-draw spread of an A8 row is 103 images, 2.62 points. Single run throughout: no repetition and no dispersion beyond that calibration spread. Every accuracy carries its k/n, and the Wilson intervals are printed for the three references only.
See Also
- FakeQuantizer API - the class, its arguments and its refusals
- Precision grammar - the widths and axes fasterai names, and which backends run them
- Quantizer - the backends that produce a model that is actually smaller
- Quantization Methods Compared - size and latency of real quantized models
- Deployable Export - exporting a quantized model and checking what came out
- BN Folding - folding batch norm into the convolution ahead of it