Eval

Per-image evaluation, Wilson intervals and paired deltas

Overview

An accuracy published alone is not comparable to anything. This harness keeps the per-image result, so two models evaluated on the same images can be compared as a paired sample instead of two lonely numbers.

Function	Criterion it answers
correct_vector, wilson	top-1: which images this model gets right, and how wide the interval around k/n is
paired_delta	top-1 again: is the difference with the reference distinguishable from zero
params	size: how many weights, packed quantized weights included
peak_activation_bytes	memory: the peak of live activations for one image
macs	compute: multiply-accumulates for one image
agreement	do two artifacts of the same model predict the same class (a parity check, not a card criterion)

A NaN logit, an empty dataloader or a model left in training mode raises: a silent wrong number is worse than a stack trace. The harness never moves the model, only the batches — the model must already be on device. Anything callable on a batch is scored, a TorchScript module and an ONNX session included; macs and peak_activation_bytes read a nn.Module or an FX GraphModule, and say so when they cannot.

---

source

predictions

def predictions(
    model:Callable, # a model, or anything callable on a batch of inputs
    dl, # dataloader yielding (inputs, targets)
    device:str | torch.device='cpu', # device the batches are moved to; the model must already live there
)->np.ndarray:

Predicted class of every image, in dataloader order

---

source

correct_vector

def correct_vector(
    model:Callable, # a model, or anything callable on a batch of inputs
    dl, # dataloader yielding (inputs, targets)
    device:str | torch.device='cpu', # device the batches are moved to; the model must already live there
)->np.ndarray:

Per-image correctness, in dataloader order

---

source

wilson

def wilson(
    k:int, # images classified correctly
    n:int, # images evaluated
    z:float=1.96, # 1.96 for a 95 % interval
)->tuple[float, float]:

Wilson score interval of an accuracy, as fractions

---

source

paired_delta

def paired_delta(
    a:np.ndarray, # per-image correctness of the reference
    b:np.ndarray, # per-image correctness of the model compared to it
    n_boot:int=2000, # bootstrap resamples over the images
    seed:int=0, # seed of the resampling
)->PairedDelta:

Accuracy difference b - a in points, with a paired bootstrap interval and the exact McNemar p

---

source

PairedDelta

def PairedDelta(
    delta:float, lo:float, hi:float, p_mcnemar:float, n:int
)->None:

Accuracy difference between two models evaluated on the same images, in percentage points

---

source

agreement

def agreement(
    pred_a:np.ndarray, # predicted classes of one artifact
    pred_b:np.ndarray, # predicted classes of the other
)->float:

Fraction of images on which two artifacts predict the same class

---

source

params

def params(
    model:nn.Module, # the model to weigh
)->int:

Number of weights; a quantized module keeps its weight packed outside parameters(), so it is counted apart (its bias is not)

---

source

macs

def macs(
    model:nn.Module, # the model to count, in eval mode
    sample:torch.Tensor, # a batch; only its first image is used
)->int:

Multiply-accumulates of one forward at batch 1, over convolutions and linear layers only — normalisation, activations, pooling and additions are not counted

---

source

peak_activation_bytes

def peak_activation_bytes(
    model:nn.Module, # the model to measure, in eval mode
    sample:torch.Tensor, # a batch; only its first image is used
)->int:

Peak bytes of the activations alive at once during one forward at batch 1; weights are size, not memory of work, so they are not counted

---

Usage

from fastermodels import correct_vector, wilson, paired_delta, predictions, agreement

ref = correct_vector(source_model, valid_dl)
opt = correct_vector(FasterModel.from_pretrained('artifact'), valid_dl)

k, n = int(opt.sum()), opt.size
wilson(k, n)                 # (0.9298, 0.9448) — the interval that belongs next to k/n
paired_delta(ref, opt)       # PairedDelta(delta=-0.51, lo=-1.2, hi=0.2, p_mcnemar=0.12, n=3925)
agreement(predictions(source_model, valid_dl), predictions(reloaded, valid_dl))

paired_delta compares the two vectors image by image, so it reads a small difference the two intervals would leave undecided. lo > floor is the publication condition; delta alone is not.

The three other criteria are read off the model itself, for one image:

from fastermodels import params, macs, peak_activation_bytes

params(fm)                             # 8_900_000 weights
macs(fm, sample)                       # multiply-accumulates for one image
peak_activation_bytes(fm, sample)      # bytes of activations alive at once

macs counts convolutions and linear layers, nothing else. peak_activation_bytes walks the FX graph and keeps a live set, so an in-place operation reuses its input’s buffer instead of adding one; a model that cannot be traced raises rather than reporting 0.

---

See Also

• Model - the model these numbers are measured on
• Card - where k, n, the interval and the delta are written down
• Gate - the condition that reads lo against the floor
• Measurement - where an on-device accuracy records its interval and ci_method

Tests live in nbs/tests/test_eval.ipynb.