Quantization Methods Compared

Size and latency of six configurations, one of them a fasterai Quantizer call, on a CPU and on a GPU, plus two Quantizer calls the library refuses on a convolutional model

Overview

This notebook times five configurations on a CPU model and five on a GPU model, and reports their size and latency. One of them comes from fasterai’s Quantizer (x86/static); the others are the untouched baseline, .half(), torchao’s quantize_ called directly, and torch.compile on the GPU FP16 model. Two further Quantizer configurations appear as refusals, called three times (twice on the CPU model, once on ResNet-50): torchao’s int8_weight_only and int8_dynamic recipes rewrite nn.Linear layers only, so on a convolutional model they would quantize the classifier head alone, which the library refuses. Quantizer backends and methods not run here (pt2e, qnnpack, fbgemm, onednn, method='dynamic', method='qat') are out of scope. The training table below is the baseline’s validation accuracy on the 1478 validation images, single run, no seed fixed; no quantized model’s accuracy is measured on this page.

• CPU section: ResNet-18 fine-tuned for 3 epochs on Oxford Pets at 64x64, timed on one training batch (dls.one_batch(), at ImageDataLoaders.from_name_func’s default bs=64; the page does not print the batch shape).
• GPU section: torchvision ResNet-50 with random weights (weights=None) on a random batch of 32 images at 224x224, on the GPU that the page names below.

Protocol for both sections: each configuration is timed in one block of 50 iterations after 10 warm-up iterations (3 for the torch.compile row), wall-clock time.time(), with torch.cuda.synchronize() around the GPU blocks; the blocks run once each, in the order shown, baseline first. No dispersion, no interleaved comparison and no output-parity check against the FP32 model is reported. Sizes are the state_dict files on disk, in MB of 10^6 bytes; the compiled model’s size is not measured.

import torch, torch.nn as nn, time
from torchvision.models import resnet18
from copy import deepcopy
from fastai.vision.all import *
from fasterai.quantize.quantizer import Quantizer

# Setup
path = untar_data(URLs.PETS)
files = get_image_files(path/"images")
def label_func(f): return f[0].isupper()
dls = ImageDataLoaders.from_name_func(path, files, label_func, item_tfms=Resize(64))

learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()
learn.fit(3)

model = learn.model.cpu().eval()
sample = dls.one_batch()[0].cpu()
print(f"Baseline model: {sum(p.numel() for p in model.parameters()):,} params")

epoch	train_loss	valid_loss	accuracy	time
0	0.611329	0.402717	0.814614	00:02
1	0.404421	0.462284	0.820027	00:02
2	0.275230	0.339791	0.849797	00:02

Baseline model: 11,704,896 params

import platform, torchao

def cpu_name():
    "The CPU model line Linux reports, or `platform.processor()` where there is none"
    try:
        for line in open('/proc/cpuinfo'):
            if line.startswith('model name'): return line.split(':', 1)[1].strip()
    except OSError: pass
    return platform.processor()

print(f"CPU: {cpu_name()} | torch threads: {torch.get_num_threads()} | "
      f"torch {torch.__version__} | torchao {torchao.__version__}")
print(f"validation set: {len(dls.valid_ds)} images")

CPU: Intel(R) Core(TM) i9-14900KS | torch threads: 24 | torch 2.9.1+cu128 | torchao 0.16.0
validation set: 1478 images

# Helper: measure size, latency, accuracy
import tempfile, os

def measure(m, name, sample=sample, n_runs=50):
    m.eval()
    # Size
    tmp = tempfile.mktemp(suffix='.pt')
    torch.save(m.state_dict(), tmp)
    size_mb = os.path.getsize(tmp) / 1e6
    os.remove(tmp)
    
    # Latency
    x = sample.to(dtype=next((p.dtype for p in m.parameters()), torch.float32))
    with torch.no_grad():
        for _ in range(10): m(x)  # warmup
        t0 = time.time()
        for _ in range(n_runs): m(x)
        latency = (time.time() - t0) / n_runs * 1000
    
    print(f"{name:30s}  size={size_mb:6.1f} MB  latency={latency:6.2f} ms")
    return {'name': name, 'size_mb': size_mb, 'latency_ms': latency}

1. Baseline (FP32)

No quantization: the fine-tuned model as trained.

results = []
results.append(measure(model, "FP32 (baseline)"))

FP32 (baseline)                 size=  46.9 MB  latency= 26.74 ms

2. FP16 (half precision)

.half() on the whole model. The file halves. The latency is 644 ms against 26.74 ms for FP32, so on this CPU this row is a size result, not a speed one. The GPU section times the same .half() model at 4.24 ms against 7.67 ms for its FP32 baseline, so the 644 ms is a property of this CPU path, not of half precision; the page does not measure why that path is slow.

model_fp16 = deepcopy(model).half()
results.append(measure(model_fp16, "FP16 (half)", sample=sample.half()))

FP16 (half)                     size=  23.5 MB  latency=644.42 ms

3. W8A8 static (x86 backend)

Weights and activations in INT8, with a calibration pass over dls.valid. At 16.78 ms against the section-1 FP32 baseline’s 26.74 ms, on the same box and the same batch, it is the CPU row with the lowest latency in this run, at a quarter of the size. With one timing block per configuration and no dispersion reported, the page cannot say how much of that difference is run-to-run variation.

model_w8a8 = Quantizer(backend='x86', method='static').quantize(deepcopy(model), dls.valid)
results.append(measure(model_w8a8, "W8A8 static (x86)"))

W8A8 static (x86)               size=  11.9 MB  latency= 16.78 ms

4. W8A32 weight-only (torchao)

Only the weights are INT8; activations stay FP32; no calibration. torchao’s int8_weight_only recipe rewrites nn.Linear layers, not convolutions (see the Quantizer reference): on this ResNet it would quantize the classifier head and leave every convolution in floating point, so Quantizer refuses the model rather than handing back a float model carrying a quantized model’s provenance. torchao rewrote none of the 20 convolutions, and Quantizer inspects the model it got back rather than trusting the recipe.

try:
    Quantizer(backend='torchao', method='int8_weight_only').quantize(deepcopy(model))
except ValueError as e:
    print('refused:', e)

refused: backend='torchao' int8_weight_only left 20 convolution(s) in floating point (0.0, 0.4.0.conv1, 0.4.0.conv2, 0.4.1.conv1, 0.4.1.conv2, 0.5.0.conv1, ...): for convolutions use backend='pt2e' or backend='x86'.

5. W8A8 dynamic (torchao)

INT8 weights, activations quantized on the fly at run time; no calibration. int8_dynamic rewrites nn.Linear layers too, so the same post-condition refuses the same model, and the message counts the convolutions the recipe would have left in floating point.

try:
    Quantizer(backend='torchao', method='int8_dynamic').quantize(deepcopy(model))
except ValueError as e:
    print('refused:', e)

refused: backend='torchao' int8_dynamic left 20 convolution(s) in floating point (0.0, 0.4.0.conv1, 0.4.0.conv2, 0.4.1.conv1, 0.4.1.conv2, 0.5.0.conv1, ...): for convolutions use backend='pt2e' or backend='x86'.

6. W4A32 (INT4 weight-only)

Weights in 4 bits through torchao’s IntxWeightOnlyConfig, applied to Conv2d and Linear layers. The default intx_packing_format is UNPACKED_TO_INT8, so each 4-bit value occupies one byte on disk: the file is about four times smaller here, not eight. On this CPU it came out at 26.67 ms against the FP32 baseline’s 26.74 ms, a difference one timing block cannot resolve.

from torchao.quantization import quantize_, IntxWeightOnlyConfig

model_w4 = deepcopy(model)
quantize_(model_w4, IntxWeightOnlyConfig(weight_dtype=torch.int4),
          filter_fn=lambda mod, fqn: isinstance(mod, (nn.Conv2d, nn.Linear)))
results.append(measure(model_w4, "W4A32 (INT4 weight-only)"))

W4A32 (INT4 weight-only)        size=  12.0 MB  latency= 26.67 ms

7. W4A16 (INT4 weights, FP16 activations)

The INT4 model above with activations in half precision. Nearly the same size as W4A32, 11.9 MB against 12.0 MB; on CPU its latency is in the same range as the half-precision row of section 2, 657 ms against 644 ms, and on the GPU the same configuration runs at 5.09 ms against 7.67 ms.

model_w4a16 = deepcopy(model).half()
quantize_(model_w4a16, IntxWeightOnlyConfig(weight_dtype=torch.int4),
          filter_fn=lambda mod, fqn: isinstance(mod, (nn.Conv2d, nn.Linear)))
results.append(measure(model_w4a16, "W4A16 (INT4 + half)", sample=sample.half()))

W4A16 (INT4 + half)             size=  11.9 MB  latency=657.45 ms

GPU benchmarks

The same methods on ResNet-50 with random weights (weights=None) and a random batch of 32 images at 224x224, plus torch.compile on the FP16 model. The torchao weight-only call is here too, and Quantizer refuses ResNet-50 for the reason section 4 gives. Latency only: with random weights there is no accuracy to compare and no parity check against a trained reference. The compiled model’s size is not measured, and its row says so; the compiled row’s speedup is against the FP32 eager baseline and mixes two changes; the FP16 row isolates the precision one.

if torch.cuda.is_available():
    device = torch.device('cuda')
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    
    # ResNet-50, batch=32 — realistic workload that saturates the GPU
    from torchvision.models import resnet50 as tv_resnet50
    gpu_model = tv_resnet50(weights=None).eval()
    gpu_sample = torch.randn(32, 3, 224, 224)
    
    def measure_gpu(m, name, sample_in=None, n_runs=50):
        if sample_in is None: sample_in = gpu_sample
        m = m.to(device).eval()
        x = sample_in.to(device, dtype=next((p.dtype for p in m.parameters()), torch.float32))
        
        import tempfile, os
        tmp = tempfile.mktemp(suffix='.pt')
        torch.save(m.cpu().state_dict(), tmp)
        size_mb = os.path.getsize(tmp) / 1e6
        os.remove(tmp)
        m = m.to(device)
        
        with torch.no_grad():
            for _ in range(10): m(x)
            torch.cuda.synchronize()
            t0 = time.time()
            for _ in range(n_runs): m(x)
            torch.cuda.synchronize()
            latency = (time.time() - t0) / n_runs * 1000
        
        print(f"{name:35s}  size={size_mb:6.1f} MB  latency={latency:6.2f} ms")
        return {'name': name, 'size_mb': size_mb, 'latency_ms': latency}
    
    gpu_results = []
    
    # FP32
    gpu_results.append(measure_gpu(deepcopy(gpu_model), "FP32 (baseline)"))
    
    # FP16
    gpu_results.append(measure_gpu(deepcopy(gpu_model).half(), "FP16 (half)",
                                   sample_in=gpu_sample.half()))
    
    # W8A32 torchao: refused here too, for the reason section 4 gives
    try:
        Quantizer(backend='torchao', method='int8_weight_only').quantize(deepcopy(gpu_model))
    except ValueError as e:
        print('refused:', e)
    
    # W4A32
    from torchao.quantization import quantize_, IntxWeightOnlyConfig
    m_w4 = deepcopy(gpu_model)
    quantize_(m_w4, IntxWeightOnlyConfig(weight_dtype=torch.int4),
              filter_fn=lambda mod, fqn: isinstance(mod, (nn.Conv2d, nn.Linear)))
    gpu_results.append(measure_gpu(m_w4, "W4A32 (INT4 weight-only)"))
    
    # W4A16
    m_w4fp16 = deepcopy(gpu_model).half()
    quantize_(m_w4fp16, IntxWeightOnlyConfig(weight_dtype=torch.int4),
              filter_fn=lambda mod, fqn: isinstance(mod, (nn.Conv2d, nn.Linear)))
    gpu_results.append(measure_gpu(m_w4fp16, "W4A16 (INT4 + half)",
                                   sample_in=gpu_sample.half()))
    
    # FP16 + torch.compile
    import logging
    logging.getLogger("torch._inductor").setLevel(logging.ERROR)
    m_comp = torch.compile(deepcopy(gpu_model).half().to(device), mode='max-autotune')
    x_comp = gpu_sample.half().to(device)
    with torch.no_grad():
        for _ in range(3): m_comp(x_comp)  # compile warmup
        torch.cuda.synchronize()
        t0 = time.time()
        for _ in range(50): m_comp(x_comp)
        torch.cuda.synchronize()
        comp_lat = (time.time() - t0) / 50 * 1000
    gpu_results.append({'name': 'FP16 + torch.compile', 'size_mb': 'not measured', 'latency_ms': comp_lat})
    print(f"{'FP16 + torch.compile':35s}  size= not measured  latency={comp_lat:6.2f} ms")
    
    import pandas as pd
    df_gpu = pd.DataFrame(gpu_results)
    df_gpu['speedup'] = df_gpu['latency_ms'].iloc[0] / df_gpu['latency_ms']
    print()
    print(df_gpu[['name', 'size_mb', 'latency_ms', 'speedup']].to_string(index=False))
else:
    print("No GPU available — skip GPU benchmarks")

GPU: NVIDIA GeForce RTX 5090
FP32 (baseline)                      size= 102.5 MB  latency=  7.67 ms
FP16 (half)                          size=  51.3 MB  latency=  4.24 ms
refused: backend='torchao' int8_weight_only left 53 convolution(s) in floating point (conv1, layer1.0.conv1, layer1.0.conv2, layer1.0.conv3, layer1.0.downsample.0, layer1.1.conv1, ...): for convolutions use backend='pt2e' or backend='x86'.
W4A32 (INT4 weight-only)             size=  26.4 MB  latency=  8.40 ms
W4A16 (INT4 + half)                  size=  26.0 MB  latency=  5.09 ms
FP16 + torch.compile                 size= not measured  latency=  2.22 ms

                    name      size_mb  latency_ms  speedup
         FP32 (baseline)   102.542935    7.673540 1.000000
             FP16 (half)    51.322647    4.240093 1.809757
W4A32 (INT4 weight-only)    26.376239    8.398938 0.913632
     W4A16 (INT4 + half)    26.040111    5.087900 1.508194
    FP16 + torch.compile not measured    2.222867 3.452091

Comparison

CPU section, from the results list above.

import pandas as pd

df = pd.DataFrame(results)
df['size_reduction'] = df['size_mb'].iloc[0] / df['size_mb']
df['speedup'] = df['latency_ms'].iloc[0] / df['latency_ms']

print(df[['name', 'size_mb', 'latency_ms', 'size_reduction', 'speedup']].to_string(index=False))

                    name   size_mb  latency_ms  size_reduction  speedup
         FP32 (baseline) 46.912607   26.741333        1.000000 1.000000
             FP16 (half) 23.477471  644.418526        1.998197 0.041497
       W8A8 static (x86) 11.870623   16.781878        3.951992 1.593465
W4A32 (INT4 weight-only) 12.047031   26.665263        3.894122 1.002853
     W4A16 (INT4 + half) 11.918135  657.445998        3.936237 0.040675

Summary

The figures are the ones printed above, rounded to the precision shown; this table restates them.

Configuration	CPU: ResNet-18, one training batch at 64x64	GPU: ResNet-50, batch of 32 at 224x224
FP32	46.9 MB, 26.74 ms	102.5 MB, 7.67 ms
FP16 (.half())	23.5 MB, 644 ms	51.3 MB, 4.24 ms
W8A8 static (x86)	11.9 MB, 16.78 ms	not run
W4A32 (INT4 weight-only)	12.0 MB, 26.67 ms	26.4 MB, 8.40 ms
W4A16 (INT4 + half)	11.9 MB, 657 ms	26.0 MB, 5.09 ms
FP16 + torch.compile	not run	size not measured, 2.22 ms

Measured on: CPU section, the Intel(R) Core(TM) i9-14900KS printed above, with 24 torch threads, torch 2.9.1+cu128 and torchao 0.16.0; ResNet-18 fine-tuned 3 epochs on Oxford Pets, one training batch at 64x64 (default bs=64, shape not printed). GPU section, NVIDIA GeForce RTX 5090; ResNet-50 with random weights, random 32x3x224x224 input. Each configuration: one block of 50 iterations after 10 warm-ups (3 for the compiled row), wall-clock time, run once, no dispersion, no interleaving, no parity check. Nothing here was measured on other hardware, batch sizes or models.

Tool	Used for
Quantizer(backend='x86', method='static')	W8A8 static with calibration
Quantizer(backend='torchao', method='int8_weight_only')	refused on both models here: torchao rewrites Linear layers only
Quantizer(backend='torchao', method='int8_dynamic')	refused on the CPU model for the same reason
torchao quantize_ with IntxWeightOnlyConfig(weight_dtype=torch.int4)	W4A32 and W4A16

---

See Also

• Quantizer API - backends, methods and the precision grammar
• Deployable Export - exporting a quantized model and checking what was produced
• Walkthrough - end-to-end optimization pipeline