Gate

The ten conditions an artifact meets before it is published

Overview

The gate reads the artifact that was produced, not the intentions of the script that produced it: it reloads the weights in a fresh interpreter, reads the exported ONNX back, and reads the card back. The reload goes through load, so a variant published in INT8 only — a TorchScript file and no safetensors — is reloaded, run on one zero image to prove it executes, and its file digest compared to the manifest.

# Condition Passes when
0 license the license has an id and the name that validated it
1 fresh-interpreter reload a scrubbed subprocess reloads the form the artifact publishes, to the published hash
2 parity every same-precision arm agrees on at least 99.5 % of the images, and every batch-invariance arm differs by at most BATCH_TOL; cross-precision arms are recorded and printed, not thresholded
3 accuracy delta measured the paired delta carries a finite interval; whether its target is demonstrated is reported, not judged. An INT8 variant also carries, on every row, the quantization step measured against the artifact’s own FP32 form
4 exported file the ONNX the manifest claims is there, with opset 17, a dynamic batch and its Q/DQ pairs
5 head and widths the head has the expected number of classes and the widths are recorded
6 size, memory, MACs and latency every row carries size, memory and MACs as non-negative ints, and there are latency rows or the not measured sentinel (the old spelling non mesurée is still read for one release)
7 card the README is there and check_card flags nothing
8 clean-machine reload a clean machine reloaded the artifact and got the same result
9 on-target claim latency rows come with a proof measured on the target

The gate guards the integrity of the measurement, not the size of the effect: condition 3 asks that the delta was measured with a finite interval and reports whether the target is demonstrated — the lower bound clears it — or not demonstrated, which says something about the evidence and not about the model. A variant whose interval does not clear its target is publishable: more compressed variants of one model serve more constrained uses, and the card says where each one lands.

A number that is missing is not a pass: condition 3 fails when there is no lo to read, and condition 6 fails naming the criterion a row does not carry.

BATCH_TOL is 1e-3 because a float32 CPU model is not bit-exact across batch sizes: measured on this harness, resnet18 differs by 4.29e-06 between batch 1 and batch 32 and mobilenet_v3_large by 1.24e-05 (the convolution path depends on the batch size), while the same models exported to TorchScript INT8 differ by exactly 0. What the condition catches — a BatchNorm left in training mode, an export pinned to one batch size — moves the outputs by order 1.

source

run_gate

def run_gate(
    artifact_dir:str | Path, # directory holding config.json, model.safetensors and README.md
    manifest:dict, # what the producer claims about the artifact
    *, python:str | None=None, # interpreter of the fresh-interpreter reload (default: this one)
    pythonpath:str | None=None, # the only PYTHONPATH that reload is given
)->list[GateRow]:

Run the ten publication conditions on a local artifact directory

source

GateRow

def GateRow(
    condition:int, name:str, passed:bool, evidence:str
)->None:

One publication condition and what it found

source

gate_passed

def gate_passed(
    rows:list[GateRow], # what `run_gate` returned
)->bool:

True when every condition passed

Usage

from fastermodels import run_gate, gate_passed

rows = run_gate('artifact', manifest, python=sys.executable, pythonpath='/path/to/fastermodels')
for r in rows: print(f"{r.condition} {r.name:26} {'pass' if r.passed else 'FAIL'}  {r.evidence}")
gate_passed(rows)
0 license                    pass  id='bsd-3-clause' validated by 'nathan'
1 fresh-interpreter reload   pass  reloaded 6f3c... vs manifest 6f3c...
2 parity                     pass  ['in-memory', 'safetensors'] 1.0; ['batch 1', 'batch 32'] max_abs_diff=1e-07
3 accuracy delta measured    pass  arms=INT8 ONNX vs reference delta=-0.51 lo=-1.2 hi=0.2 target=-2.0 met
6 size, memory, MACs and ... pass  1 rows measured, latency_rows='not measured'
...

The reload of condition 1 runs with PYTHONPATH set to that one path and nothing else, so it fails if the artifact only reloads thanks to something in the producer’s environment.

See Also

Tests live in nbs/tests/test_gate.ipynb.