Tutorial contract

What a fasterai tutorial may claim, and the checks that enforce it

A fasterai tutorial is a claim about what the library does to a real model. This page is the contract those claims are held to, and it is also the enforcement: the hidden cells below parse every tutorial in nbs/tutorials/ (plus quickstart and overview) and assert the rules that follow. They run in the default test suite, so a page that breaks the contract turns CI red before it reaches the site.

The voice

A tutorial reports what one run measured. It does not rank, advise or promise.

Instead of Write
“the best criteria for this” “in this run, large_final ended at 92.69% (1370/1478)”
“always fold batch norm before export” “folding rewrites the convolution weights; this page folds before exporting”
“up to 4x faster” “633 ms against 26 ms on this box, single run”
“minimal accuracy loss” “-3.1 points, and the two Wilson intervals overlap”

These words never appear in tutorial prose, because each of them states a preference or a guarantee that no cell on the page measured: best, fastest, always, never, recommended, rule of thumb, key finding, tip, guaranteed, optimal, when to use, for free, free speedup, no drop in accuracy, without any drop, and the verdict emoji. Code is not prose: warnings.simplefilter('always') is fine, and so is a file named best.pt.

Every number comes from the page

The rule is mechanical: a number stated in prose must be a number a cell on the same page prints, exactly or as a rounding of it. A percentage may be printed as a proportion (0.926928 backs 92.69%), a parameter count may be printed in full (134309962 backs 134 million). What is not allowed is a number nobody computed: a ratio, a delta or a speed-up that only exists in the sentence. If a page wants to say “3.9x smaller”, a cell prints the ratio.

Confidence bounds are the one exception, and only because they are checked harder: an interval written next to a k/n is recomputed from k and n and has to match.

Measurement rules

  • Accuracy carries its n. A sentence stating an accuracy carries n = N, or a k/n, or names the evaluation set size (“the 1478 validation images”).
  • Percentages that come from counts are written with the counts, and the two must agree: 92.69% (1370/1478).
  • Intervals are Wilson 95% intervals and are recomputed from k/n — on whichever scale the page prints, percentages or proportions.
  • A latency carries a scope line: “single run”, “one run”, “this run” or “Measured on …”. A number with a ms, µs or FPS unit and no scope is a claim about a machine the reader does not have.
  • A sentinel is never a result. 0.00 MB, 0 parameters, nan ms, accuracy: -1 are failed measurements; they may be discussed, never shown as an outcome.

Required elements

Front matter with title, description and skip_exec: true; a ## Summary section; a ## See Also section; every link resolving to a page or an image that exists. tutorials/quantize/deployable_export.ipynb is the reference for the shape of a page.

The rules the checks enforce

Rule What it asserts Waivable
R1 No cell output is an error or contains a traceback no
R2 A visible cell that produces something shows it, and no output is a bare placeholder yes
R3 No sentinel value (0.00 MB, 0 parameters, nan, -1) is shown as a result no
R4 Every number in prose is printed by a cell on the same page yes
R4b A page reporting a latency carries a scope line yes
R5 The prose uses none of the forbidden words yes
R6 Names shown in prose exist: blocks parse, keyword arguments and vocabularies are real yes
R7a Front matter, ## Summary and ## See Also are present yes
R7b Every internal link resolves no
R8 The sidebar and the tutorials folder agree, both directions no
R9 Accuracy sentences carry n; k/n and intervals are recomputed yes
R10 No cell output is a stderr stream, hidden cells included yes

“Waivable” means the rule can hold an entry in the quarantine described below. The four that cannot be waived are the ones whose failure means the page is broken rather than imperfect: a failure shown as a result, and a link or sidebar entry that goes nowhere.

Running the checks

nbdev_test --path nbs/tutorial_contract.ipynb          # seconds; this is what CI runs

And, on a machine with the GPU, the datasets and the optional dependencies, the execution gate — it re-runs every tutorial in a copy of its folder and reports what the code no longer reproduces:

python -c "from nbdev.test import nbdev_test; nbdev_test(path='nbs/tutorial_contract.ipynb', flags='tutorials')"

It takes about an hour. FASTERAI_TUTORIALS='tutorials/sparse/*' narrows it. Nothing is written under nbs/, and no output is committed by it: when a page drifts, the remedy is to re-run that page in a Jupyter kernel and commit the new outputs together with the prose they support.

The quarantine, and how it empties

Pages written before this contract carry known violations. Rather than block them, the gate holds a QUARANTINE dict — a per-page list of the rules that page is still allowed to fail — and enforces three things about it:

  1. It can only shrink. A frozen copy, _LANDED, was taken when this page landed. A page or a rule that is not in _LANDED cannot enter QUARANTINE, so a new violation can never be waived — it must be fixed.
  2. It cannot hold an unwaivable rule.
  3. Entries expire the moment they are unused. If a quarantined page stops violating its rule, the gate fails with “now passes R4: delete the entry”. Rewriting a page and deleting its entry are one commit.

_LANDED is edited by review only; nothing in the machinery writes to it.

To let a single sentence through R5 — an honest caveat that happens to contain a listed word — add it verbatim to LEXICON_ALLOW[page]. The match is the whole sentence, so the diff shows exactly what was allowed.

What these checks cannot do

They read structure, not meaning. They cannot tell whether a sentence’s conclusion follows from the number beside it, whether a comparison is controlled, whether a baseline is degenerate, or whether a python block would actually run. Those stay with the reviewer of every tutorial change. Two known blind spots are worth naming: Pruner and PruneCallback accept **kwargs, so no signature can refute a made-up keyword on them (a denylist covers the ones already seen), and a cell whose last statement is a call is not required to have output, because a call may legitimately return None.

See Also