dotfiles/claude/memory/reference-verification-ladder.md

---
name: reference-verification-ladder
description: Canonical home for the proven verification instruments — the named gates
  any session reaches for instead of re-deriving. Steward-authorized consolidation
  2026-06-05 (skill-harvest ruling); ends the per-wrap re-proposing of individual
  instruments.
metadata:
  node_type: memory
  type: reference
  originSessionId: b3202a24-301f-499d-af9e-ac5368dd2c73
  modified: 2026-08-02T16:04:03.432Z
permalink: claude-memory/reference-verification-ladder
---

# The Verification Ladder — named instruments

Proven gates, each earned from a real catch. Reach for the one the claim's shape demands; a claim without its gate is an assertion. Linked disciplines: [[feedback-trust-prior-pass-frame]], [[feedback-verify-each-post-type-after-changes]], [[feedback-rank-on-fields-you-actually-write]].

## Output-equivalence claims ("this refactor changes nothing")
- **Byte-identical compile gate** — prove with `cmp`/`diff` at BOTH the direct-invocation level and the full-pipeline level. Proven 3×: SCSS Stage 1 (`c37a3bd`), site.hs listing generator, §I.k transform dormancy.
- **Expected-delta gate** — when diff=0 is impossible, prove the delta's *complete shape*: `sort -u` over all per-file diffs; every line must belong to a named class (2026-06-04).
- **Delta classification** — corpus-wide variant: classify EVERY changed line into named classes; the unclassified residue must be exactly empty-or-whitespace. Catches what sampling misses, including one's own shipping comments (decompose + SEO passes, 2026-06-05).
- **Delayer-diff** (2026-06-08) — for an `@layer`/structural-wrap refactor where the compiled output gains uniform wrappers: strip the `@layer X{`/closing-`}` wrappers from the compiled CSS (brace-aware script), diff vs the true baseline; byte-identical proves nothing *moved, dropped, or changed* — only got wrapped. Proven 3× in W3R Stage 2a. **CAUTION — byte-identical ≠ rendered-identical under `@layer`:** `@layer` overrides specificity, so a structural-wrap byte-proof is *necessary but not sufficient*; render-review is the gate when the cascade *basis* changes (a 5-layer wrap was byte-true yet produced 3 render inversions the diff could not see — the steward's eye caught them). Rendered-identity is then argued *by construction* (only already-winning rules promoted) AND render-reviewed — not inferred from the byte-diff.
- **Hold or exclude volatile build-stamps** — a build-date/commit stamp (colophon `_build_info`) differs between baseline and post builds and falsely flags a byte-diff. Hold it constant or exclude it, and say which (2026-06-10, Stage-G close).
- **Two-hat commit separation** — name the hat each commit wears: a *refactor* commit's compiled output is byte-identical (or carries a pre-stated classified delta); a *feature* commit is where rendered values change. Makes the gate above applicable per-commit instead of per-session (2026-06-08, from Fowler).
- **Lossless-relocation gate** — restructuring an index or moving prose between files: line-range **slices, never retyping**, + md5-conservation per moved slice + a link-resolution canary over every `](file.md)` and `[[wikilink]]`. Proven on the MEMORY.md two-file split (2026-07-06) and again on the skill-harvest split (2026-08-01, archive byte-identical over 166,027 bytes).

## Mechanical text transforms ("strip/replace token X across the file")
- **Token-strip hits prose** (2026-06-08) — a whole-file `sed s/…!important…//` (or any token strip) also removes the token from comments, code-refs, and docstrings, silently mangling prose. Scope the transform to declaration lines, OR compare raw-before vs clean-declaration counts and verify comment integrity after (caught 4 mangled comments via the 56-vs-70 count mismatch; restored). Same family as *Count first, then look* — the word lives in prose too.

## Coverage claims ("all X were handled")
- **Exhaustive route/rule census** — enumerate ALL routes/rules/sections before classifying any; classify each explicitly; no sampling. (The censused-routes discipline; §I.k.c measuring read.)
- **Count first, then look** — never read a census through truncation (`| head`, clipped listings). `wc -l` before inspection; 3 same-day instances 2026-06-05 (chamber/CM-AI/studium untracked files).
- **Re-audit with the TOOL's recognizer, not the classifier that shares its blind spot** — when a classifier and the tool it feeds share a predicate, the classifier's *mis*-classifications masquerade as genuine new categories. Size a gap by running the tool's own recognizer over the corpus.
- **A silent safety net is uninformative, not reassuring** — a fallback / fail-loud branch / `unrecognized` kind that never fires across N real cases has not been shown to work; census the substrate for what it should have caught. (Generalized from the negative-grep class to net-silence.)
- **Verify at the granularity of the mutation, not the aggregate** — a document-wide word-multiset guard PASSES while a per-item swap (a word from note A landing in note B) violates the intent, because the total bag is unchanged. Check per item, keyed by the item's own identity.
- **Work-in-omnibus: verify the interior, not the endpoints** — when a sidecar brackets one work out of a multi-work source by heading boundaries, sample the span INTERIOR and scan for embedded works and editorial apparatus; endpoint-verification passes over a wrong interior.

## Build/render claims ("it works / it looks right")
- **Render-and-LOOK** — render the artifact and read it before asserting; never reason from the remembered model (caught glued FR colons, feTurbulence errors, viewport misreads).
- **Per-type × per-viewport** — after any shared CSS/template change, every content type on desktop AND phone ([[feedback-verify-each-post-type-after-changes]]).
- **Compiled-selector grep gate** — after SCSS changes, grep the *compiled* CSS for retired/duplicated selectors and late overrides (caught opacity duplicates + silent font-family override, 2026-06-03).
- **Governed-char byte gate** — verify specific codepoints by byte (`[8239]` probes), not by eye; verify test INPUTS too (the `\x202Fc` hex-escape trap).
- **Measure the box model before theorizing the mechanism** — headless-Chrome measurement caught a per-point *padding asymmetry* where the theory said width (2026-06-09). Reinforced across four days; the standing harness is `/measure-render`.
- **Measure the font's true average advance before a character-count measure** — the 0.5em convention misestimated EB Garamond by 32% (0.377em measured); measure over real corpus prose including spaces (2026-06-09).
- **Element shot = `scrollIntoView({block:'center'})` + full-viewport, never computed clip** — box-clip math off `getBoundingClientRect` kept landing on page-top and burned several shots (2026-06-13).
- **CSS mask fills WHITE, not black** — a `mask`/`-webkit-mask` SVG must fill the shape opaque; a black fill renders BLANK (luminance-vs-alpha). ~6 debug rounds (2026-06-13).
- **Live-CSS-patch in `_site` for in-browser iteration** — `sed` the value in the built CSS and refresh to let the steward judge sizes in the real browser, then lock the chosen value in source SCSS. Turned a multi-round sizing decision into one sitting.

## Toolchain claims ("the parser/build does X")
- **Measure-toolchain-before-spec** — probe actual behavior (ghci/REPL probes) before writing spec text that presumes it («» parse as Str; BulletList misparse; 2026-06-04/05).
- **Feature-detect gates can lie** — `@supports(animation-timeline:view())` returns true on iOS Safari 26 where mobile WebKit will not drive it. A capability probe reports the parser, not the behaviour; verify on the target (2026-06-12).
- **Parse-and-validate an artifact before its first machine consumption** — a hand-authored YAML/config that humans have only ever *read* is unvalidated input the moment code consumes it. All four chamber reading-index YAMLs failed on first parse (2026-06-12).

## Numeric/accessibility claims
- **Compute, don't eyeball** — WCAG contrast via calculator (1.84:1 vs an assumed "fine"; 2026-06-03).
- **CI-upper-bound + drop-one robustness — STANDARD for every ESCALATE** (jurist-ruled 2026-07-04): grade on the one-sided 90% Clopper–Pearson **upper** bound, never the point estimate, and re-check with any one confirmed case dropped. A point estimate at n=40 conceals how little it constrains.

## Remote/persistence claims ("it's backed up / deployed")
- **Fresh-clone gate** — verify a remote by independent clone and content check, never by push exit-code (chamber-library LFS: the remote held stubs for months while pushes "succeeded"; 2026-06-05).
- **Live-fetch gate** — verify deploys by fetching the live URL and reading bytes (robots.txt, 403 differentials; 2026-06-05).
- **Cross-volume verify-before-delete** — `rsync -a` → exact file-count match + `du` (never a byte-sum; mine computed 0=0 vacuously) + rc=0 → *only then* `rm` the source. Isolate flaky-remote pushes so a hang cannot mask a failure.

## Extension claims ("the earlier verification covers this too")
- **Re-verify at the extension's scope** — the prior pass tested what it tested ([[feedback-trust-prior-pass-frame]]; now also a Symmetria §3 flag).
- **Extending a tool re-tests its foundations** — a widen exercises shared machinery the original's tests never hit, so the extension's `--validate` must assert the *shared* invariants too. Building the block-id handler surfaced a latent duplicate-id defect (2026-07-11).
- **A stronger gate obliges a retroactive sweep** (jurist Q1b) — a stronger check existing and not pointed at what shipped under the weaker one is the exact gap the discipline exists to close.

## Causal claims about running systems ("X is what's pinning the CPU / blocking the loop")
- **Sampling-profile gate** — name the hot path by measurement (process sample / SIGUSR1→CDP CPU profile) before naming the culprit; log shape suggests mechanism but does not prove it (circle-forward read as a retry queue from log census; profile + code read showed fire-and-forget and the pin elsewhere entirely — temporal; 2026-06-06). Corollary: write the inferred-vs-measured split explicitly in any outward claim — when the profile overturns the lean, sharpening costs nothing where retraction would have.
- **Exit-code-over-tail gate** (2026-06-06): never read success from prettified command output — `git push 2>&1 | tail -1` rendered a REJECTION as the soothing fragment "...and the repository exists." Check `$?` (or let the tool surface stderr whole) before claiming a push/deploy landed. Caught same-session; the byte cost of `; echo exit=$?` is zero.
- **Verify the RUNNING BINARY's provenance, not source HEAD** — check the compiled `dist/` mtime and grep the compiled symbols before reasoning about live behaviour. A whole "schema-drift" mechanism inverted on this: HEAD carried a migration the running process did not.
- **Quantify the removed cost as the A/B control** — when a fix *removes* a hot operation, time the removed operation on real data rather than racing a flaky end-to-end. (2.56 s vs 0.003 s on the real 676k graph.)
- **Confirm a named cause by swap-in** — naming the true reference/config/cause is identification, not proof; swap the candidate in and watch the anomaly collapse, ideally to zero.
- **Split cause from magnitude before sizing a remedy** — a bucket keyed on one summary axis holds heterogeneous causes, and a single label silently implies a single fix at a single scale. Read the substrate and split first.
- **Revert-and-redo-smaller** — when a gate fails and the cause is not immediately visible, return to the last green state and redo the step smaller; do not debug forward. Named as the reflex we most lack (2026-06-08, from Fowler).

## Enforcement rules built on a "never occurs" premise ("ARC English never uses the em dash")
- **Corpus-census-the-premise gate** (2026-06-07): before an enforcement transform ships, grep the corpus for the premise's counterexamples — the §I.k.b em→en rule's "never uses it" was false for exactly one fixed convention (the Chicago `———.` bibliographic repeat, 2 carriers), and it mangled Sources blocks for three days because both the build and the audit verified the rule-as-stated, not the premise-against-the-corpus. One grep would have caught it.

## Render-gate comparisons ("the steward rules on render")
- **Element-census the comparison page** (2026-06-07): before spending the steward's eye, verify the offered page actually carries the element under judgment — the first hr-gate page offered had zero bare `<hr>` (its dots were an ornament, identical in both builds); the steward saw no difference because none could exist there.
- **Pre-registered tool trial** (2026-06-07): baseline commit before first contact (every later diff attributes cleanly) + acceptance criteria written into the commit message BEFORE any experience with the tool + diff classification + exercise-before-verdict (drive the real API against questions you genuinely need answered). Made the Basic Memory verdict ~30 min, evidence-grade, attachment-free; the instrument for every future tool trial, BMF milestones included.

## Gate-design claims ("the gate would have caught that")
*The family earned hardest and most often. A gate's silence is evidence only about what the gate can see.*
- **The gate itself can be PASS-BUT-FALSELY** — a verifier checking an *enumerated* set of signatures silently passes residue outside the set (`verify_conversion` reported `cruft=0` over 59 raw `<sup>`). Gate on the **class**, not the instances.
- **Method-class vs calibration** — when a metric fails to separate two cases, ask whether it is mis-*calibrated* or structurally *blind*. No retuning of a set metric reaches an order-only difference: the blindness is in the method class, not the threshold.
- **Coverage never attests order** — any permutation preserving local windows passes a coverage comparison, *however independent the compared extractors*: coverage discards position before comparing. Demonstrated twice on real material and ratified as constitutional text (spec v2.9.0, REVIEWED-84, 2026-07-29).
- **A word-guard passes on scrambled text** — word *presence* cannot catch reading-order damage (a two-column read across the gutter): same words, wrong order, PASS. The gate for it is reading-order coherence, not the multiset.
- **Positive test at the enforcement path, never a negative grep** — for any "can this be bypassed?" property, a grep proves the absence of a *string*. Drive a known-bad input at the real enforcement path and watch it refuse (2026-07-14, jurist-caught).
- **Read the gate's decision code before designing its consumer** — reading `verify_body_conservation.classify` revealed it classifies boundary runs by position+size only and *provably cannot* confirm class identity; that "no, and knowably no" shaped an honest design and corrected the package mid-build (2026-07-22).
- **Implement the relation, not an approximation of it** — where code implements a *ruled* relation or criterion, the acceptance path must BE the relation; a structural approximation (word-aligned matching of a string-defined relation) is a different gate wearing the ruled one's name.
- **Seam-probe the artifact — gates test claims, probes test joins** — two real defects (empty footnote defs; defs swallowed by a trim) were invisible to every word-level gate because no words were lost, only structure detached. Probe the joins (2026-07-21).
- **`cmp` after apply — a tool's report is not its write decision** — `strip_cruft --apply` prints transform counts *before* deciding to write; three distinct refusal causes each read as "applied" until `cmp` proved otherwise (2026-07-21).
- **A control must sit at the layer the defect lives in** — the PDF-origin classifier passed 8/8 while mis-classifying two real scans: every control exercised the decision *rule*, so none could test whether the *signal set* was sufficient. Ask what layer a green check occupies before reading it as coverage (2026-08-01, REVIEWED-83 A1).
- **Prose word-guard for faithful structure-cleaning** — when a cleaner reflows structure but must preserve prose, gate on a prose-only word-multiset compare (headings, numerals, printed-title runs and markup excluded).

## Grounding and citation claims
- **Per-claim citation** — verify each citation against the *specific* claim it carries; never let a sub-agent's aggregate "4/4 verified" ride (jurist-caught: a paper cited for a claim it did not support).
- **Assert "X is in Y" only after reading Y** — grounding must reach the file that HOLDS the fact, not one that describes it. Earned hard 2026-07-17: a claim about `test_tools` written from the tool's own docstring — a derived self-assessment — was false.
- **Byte-check hand classifications at source before citing them** — two of my classes *and* a jurist assessment were overturned by reading the bytes; the correction strengthened the finding (2026-07-19).
- **Reconcile against the authoritative source before any "closed/complete/finally"** — reconcile against the roadmap or the spec's own named-open items, never the derived close-list.
- **"Closable-now" is itself a claim to check** — the closable-vs-blocked partition must be reviewed, not asserted; the named-dependency discipline only protects if the closability judgment is checked too (jurist-named).
- **Implementation is a second gate** — *"a gate that passes a document and a build that has to implement it are different tests"* (jurist, after their own 07-16 miss surfaced at build orientation). A text passed by reading is re-tested by having to act on it.

## Provenance and re-anchor claims
- **Re-anchor = re-verify, by sha-match** — reconstruct the OLD bound state from git by matching the declared sha, prove the old→live delta is exactly the expected hunks, then re-verify spans. Re-anchoring without re-verifying rebinds a pointer to unexamined content.
- **Dry-run-first for bulk file operations** — any mass `mv`/graduate/rename emits the full move-map + collision guard + reference scan (`git grep` each old basename) before `--apply`. Proven 3× in one session.
- **Re-verify a workflow's per-item dispositions on the real apply** — a sub-agent's per-item verdict from a dry-run on a temp copy can differ from the real run; its own audit can disagree with itself.

## Governed-document changes
- **Governed supersession procedure** — `cp` live→`-vNEW` → *bounded edits, never retyping* → `diff` shows only intended alterations and insertions → header carries `Obsoletes` + `GROUNDED-IN` → on promote: freeze old, copy new to live, byte-prove the untouched remainder.
- **Reassigned-component check** (jurist-minted, REVIEWED-72) — *"when a change reassigns a named component, check what else names it."* F4 moved the born-digital candidate onto pandoc and silently voided a clause that named the converter.
- **Structural safety is PROVISIONAL; end-to-end proof on real material is FINAL** — no widen or transform is ruled final on structural reasoning alone. "Same risk as (a)" was true of the matching logic and silent on the rendering logic.

## Test-harness claims
- **The sandbox must pin the SAME module object the code imports** — patch `sys.modules`, not a freshly-exec'd copy; a `_load()`-style loader builds a different object and the mispinned sandbox tests nothing.

## Gate design — the discrimination gate
- **A check must discriminate between two REAL artifacts, one known to have the property and one known to lack it.** Same verdict on both = the check has demonstrated nothing, however many synthetic fixtures it passes. Earned 2026-08-02 after **four instances in three days** of a *passing* check certifying a property of the **code** while claiming a property of the **result**, every one found by a person looking: the vignette field colour bound to a class no element carried · the Fool harness recording `degraded:null` on a run with no answer · five splitter defects found only by contact with real documents · the §3.3 screen false-passing a package whose Part VII *is* a collected limitations section.
- **The reason positive controls did not catch these: the fixtures were derived from the CHECK, not from the PROPERTY.** "What makes this regex fail?" instead of "what makes this claim false?" A control built from the check's own vocabulary inherits its blind spot by construction — the same shape as *controls built by extraction leak by construction*. **Derive fixtures from the property; draw them from real artifacts**, since a synthetic negative is written by the same hand as the check.
- **Every check states, in its own output, what it did NOT establish** — the necessary-but-not-sufficient gap named beside the pass. A check that cannot name its gap does not ship.
- **The residue is irreducible and needs a differently-formed reader.** Discrimination catches proxy-gaps where a real negative instance exists; it cannot catch a proxy that discriminates on the pair and fails elsewhere. What is left must be *looked at* before the claim is made, by someone who is not the check's author — Constraint 6 applied to instruments. Reference implementation: `dotfiles/claude/governance/fool/test_discrimination.py`, which is shown rejecting the §3.3 pattern **as it actually shipped**.

## Estimates and schedules
- **Quote a long-job ETA only from an observed rate** — twice in one day I gave an ETA from intuition and was wrong by ~30×. Measure rows-per-elapsed on the running job, or benchmark a slice, then quote.

## Structure recovery
- **One placement engine, two map-producers** — chapter-structure recovery is a single engine (`insert_chapter_headings`: anchor → unique line → replace-block/insert, fail-loud, dry-run) fed by either the EPUB NCX ToC or the OCR page-number method. Anchors are a text-prefix or a literal `<span id>`.
- **Glyph-outline → SVG from a woff2** — `fontTools` `SVGPathPen` + `BoundsPen` over the woff2, y-flipped into SVG space, to build a typographic asset from the *real* glyphs rather than a trace.

---

*Batch-append 2026-08-01 discharging **Stroke 2** of the 2026-07-19 skill-harvest FULL REVIEW ("ALL earned ladder entries queued in this register … append to `reference-verification-ladder.md` with provenance, kin merged in the same pass"). **49** open ladder-kind entries were queued, not the ~25–30 the Stroke estimated; the authorization is by extension ("ALL earned"), not by count. Kin merged into existing claim-classes where one existed; seven new claim-classes added for families that had no home — the largest, **gate-design claims**, is the one this practice has earned most often. Two queued rows were NOT folded here because Stroke 3 already ruled them skills rather than ladder notes (`/measure-render`, `/clone-test-runtime-fix`); they remain build-on-need. Detail and origin for every entry: `skill-harvest-archive.md`.*