Measurement

On-device latency rows and accuracy rows, and the checks that decide whether each is well formed and valid

Overview

Speed and accuracy do not depend on the same things. How fast an engine runs depends on its graph — the architecture, the widths, the numeric precision — and on the device and software stack; it does not depend on the values of the weights. How accurate it is depends on the weights and on the evaluation set. So a measurement is one of two kinds of row, and neither needs the other:

• a latency row times one engine on one device. It needs no accuracy, so a model with random weights (weights='random') is measured like any other: this is how a latency predictor gets trained on architectures nobody has trained yet.
• a quality row measures one artifact’s accuracy on one evaluation set, through the engine that runs it.

Rows are stored as JSON Lines, one object per line, appended and never rewritten; a file may hold both kinds. FIELDS[kind] is the schema of each kind: every field’s type, whether it is required, and the rules its value obeys. A field that is not required may be absent or None, which means not recorded.

Kind	Group	Fields
both	identity and provenance	schema_version, kind, run_id, measured_at (ISO-8601, UTC), series, harness_commit, fasterai_version, report_url
latency	graph	architecture, graph_hash, input_resolution, macs, params, variant, recipe, weights (trained or random), model_id, model_revision
latency	artifact	precision, int8_mode (iff int8), engine_hash, size_mb, engine_layers, reformat_layers, layer_precisions
latency	device and runtime	device, device_host, runtime, runtime_version, platform_version, power_mode, clocks_locked, batch_size
latency	conditions	temp_start_c, temp_end_c, gpu_freq_mhz, throttled, warmup_s, duration_s, repeats, latency_spread_pct
latency	latency	latency_ms_p50, latency_ms_p90, latency_ms_p99, latency_scope, host_latency_ms, latency_source
quality	model and artifact	model_id, model_revision, model_hash, weights_license, engine_hash, precision, device
quality	accuracy	metric, eval_set, eval_set_hash, dataset_license, eval_n, accuracy, ci_low, ci_high, ci_method, measured_through, argmax_parity, parity_floor, baseline_run_id

• series is a free-text label grouping the rows measured in one session, typically one night; it is not part of any key.
• graph_hash is the sha256 of the ONNX graph with the values of every initializer removed, so two models of the same shape share it, whatever their weights. graph_hash(onnx_path) computes it.
• latency_scope says what the latency covers, e.g. engine execution only, or host transfers included; two latency rows are only comparable when it matches.
• layer_precisions counts the engine’s layers by the precision they actually ran in, e.g. {'int8': 52, 'fp16': 9}.
• engine_hash is the sha256 of the engine file that ran; model_hash is whatever fingerprint the exporter records, so it is only a string.
• A quality row names the engine it was measured through; engine_hash may be None only for an accuracy measured on the development model.

Throughput is derived, never stored: throughput(row) is batch_size * 1000 / latency_ms_p50 on a latency row.

Two questions are kept apart: validate asks whether a row is well formed, status whether the measurement it records is valid. Neither compares a row to a latency or accuracy budget: a row carries no verdict column, and whoever reads it recomputes their own. A well-formed row is INVALID when:

• latency: latency_source is not measured; throttled is true or not recorded; latency_spread_pct is above max_spread_pct (5 by default) or not recorded;
• quality: measured_through is not deployed_engine; argmax_parity is below parity_floor or not recorded.

join(latency_rows, quality_rows) pairs the VALID rows of both kinds that share an engine_hash: one dict per pair, with the fields of both. The identity and provenance fields are kept per side, as latency_run_id, quality_run_id, latency_measured_at and so on; schema_version is kept once. The fields both kinds carry (engine_hash, precision, device, model_id, model_revision) are one column: a side that did not record one takes the other’s value, and two sides that disagree on the same engine raise.

---

source

validate

def validate(
    row:dict, # one measurement row, of either kind
)->list[str]:

Every structural problem of row, each prefixed by its field name; an empty list means well formed

---

source

status

def status(
    row:dict, # one measurement row, of either kind
    max_spread_pct:float=5.0, # largest (max-min)/median latency spread over repeats, in percent
)->tuple[str, list[str]]:

(‘VALID’, []) if row is a usable measurement, else (‘INVALID’, reasons); not a verdict against any budget

---

source

key

def key(
    row:dict, # one well-formed measurement row
)->tuple:

The identity a row is unique on, its kind first; a software update on the device gives a new latency key

---

source

throughput

def throughput(
    row:dict, # one latency row
)->float:

Images per second at the timed batch, derived from the median latency and never stored

A latency row is unique on its graph, its weights (trained or random), precision, batch size, device, software stack and date; a quality row on its model, revision, engine, evaluation set, metric and date. Uniqueness is checked per kind.

---

source

join

def join(
    latency_rows:list[dict], # latency rows; rows of the other kind and invalid rows are skipped
    quality_rows:list[dict], # quality rows; rows of the other kind and invalid rows are skipped
)->list[dict]:

One dict per pair of VALID latency and quality rows sharing an engine_hash

---

source

graph_hash

def graph_hash(
    onnx_path:str | Path, # an ONNX file
)->str:

sha256 of the ONNX graph with every initializer’s values removed and its name, shape and type kept

Initializer names are part of the graph: renaming a weight gives a new hash, as it does to the engine a runtime builds from it. Opset versions are hashed too. graph_hash needs the optional onnx extra: pip install 'fastermodels[onnx]'.

---

source

read_rows

def read_rows(
    path:str | Path, # a JSON Lines file of measurement rows, of either kind
)->list[dict]:

The rows of path, as stored; status judges them

---

source

write_row

def write_row(
    path:str | Path, # a JSON Lines file, created if absent
    row:dict, # the measurement row to append
):

Append row to path; refuses a malformed row and a row whose key is already in the file

---

source

to_parquet

def to_parquet(
    rows:list[dict], # well-formed measurement rows of one kind, e.g. from `read_rows`
    path:str | Path, # the Parquet file to write
):

Write rows as one Parquet table with an explicit schema: one typed column per field of their kind

to_parquet writes one kind per file, with an explicit Arrow schema derived from FIELDS: str is a string column, int int64, float float64, bool bool, a dict a string holding canonical JSON, and a field that is not required is nullable. A missing value is a null in the file, whatever reads it. It needs the optional parquet extra: pip install 'fastermodels[parquet]'.

---

Usage

A latency row, here for an engine built from a model with random weights:

from fastermodels import measurement

lat = dict(schema_version=1, kind='latency', run_id='2026-09-28-a', measured_at='2026-09-28T10:00:00Z',
           harness_commit='deadbee', architecture='resnet50', graph_hash=measurement.graph_hash('model.onnx'),
           input_resolution=224, macs=2_900_000_000, params=17_000_000, variant='p30', recipe={'prune': 0.30},
           weights='random', precision='fp16', engine_hash='0' * 64, size_mb=34.1,
           layer_precisions={'fp16': 61}, device='jetson-nano-4gb', device_host='own', runtime='tensorrt',
           runtime_version='8.2.1', platform_version='jetpack-4.6.1', batch_size=1, throttled=False,
           repeats=5, latency_spread_pct=1.2, latency_ms_p50=2.1,
           latency_scope='engine execute, host transfers excluded', latency_source='measured')

measurement.validate(lat)                 # [] : well formed
measurement.status(lat)                   # ('VALID', [])
measurement.throughput(lat)               # 476.19... images per second, derived
measurement.write_row('rows.jsonl', lat)  # appends one line; a second call with the same row is refused

A quality row, and the join of the two kinds on the engine they share:

qual = dict(schema_version=1, kind='quality', run_id='2026-09-28-b', measured_at='2026-09-28T11:00:00Z',
            harness_commit='deadbee', model_id='org/resnet50-p30', model_revision='a1b2c3d',
            weights_license='apache-2.0', engine_hash='1' * 64, precision='fp16', device='jetson-nano-4gb',
            metric='top1', eval_set='imagenet-val', eval_set_hash='sha256:...', dataset_license='custom',
            eval_n=50_000, accuracy=78.9, ci_low=78.5, ci_high=79.3, ci_method='wilson',
            measured_through='deployed_engine', argmax_parity=99.7, parity_floor=99.0)

rows = measurement.read_rows('rows.jsonl')
measurement.join(rows, rows)              # each kind is picked from the list it belongs to
measurement.to_parquet([r for r in rows if r['kind'] == 'latency'], 'latency.parquet')

---

See Also

• Model - the published model a quality row measures
• Eval - wilson, the interval ci_method='wilson' refers to, and agreement
• Card - where a published latency is written down
• Gate - the conditions an artifact meets before it is published

Tests live in nbs/tests/test_measurement.ipynb.