dotfiles/claude/memory/reference-verification-ladder.md

name, description, metadata, permalink

name	description	metadata	permalink
reference-verification-ladder	Canonical home for the proven verification instruments — the named gates any session reaches for instead of re-deriving. Steward-authorized consolidation 2026-06-05 (skill-harvest ruling); ends the per-wrap re-proposing of individual instruments.	node_type type originSessionId modified memory reference b3202a24-301f-499d-af9e-ac5368dd2c73 2026-08-02T16:04:03.432Z	claude-memory/reference-verification-ladder

The Verification Ladder — named instruments

Proven gates, each earned from a real catch. Reach for the one the claim's shape demands; a claim without its gate is an assertion. Linked disciplines: feedback-trust-prior-pass-frame, feedback-verify-each-post-type-after-changes, feedback-rank-on-fields-you-actually-write.

Output-equivalence claims ("this refactor changes nothing")

• Byte-identical compile gate — prove with cmp/diff at BOTH the direct-invocation level and the full-pipeline level. Proven 3×: SCSS Stage 1 (c37a3bd), site.hs listing generator, §I.k transform dormancy.
• Expected-delta gate — when diff=0 is impossible, prove the delta's complete shape: sort -u over all per-file diffs; every line must belong to a named class (2026-06-04).
• Delta classification — corpus-wide variant: classify EVERY changed line into named classes; the unclassified residue must be exactly empty-or-whitespace. Catches what sampling misses, including one's own shipping comments (decompose + SEO passes, 2026-06-05).
• Delayer-diff (2026-06-08) — for an @layer/structural-wrap refactor where the compiled output gains uniform wrappers: strip the @layer X{/closing-} wrappers from the compiled CSS (brace-aware script), diff vs the true baseline; byte-identical proves nothing moved, dropped, or changed — only got wrapped. Proven 3× in W3R Stage 2a. CAUTION — byte-identical ≠ rendered-identical under @layer: @layer overrides specificity, so a structural-wrap byte-proof is necessary but not sufficient; render-review is the gate when the cascade basis changes (a 5-layer wrap was byte-true yet produced 3 render inversions the diff could not see — the steward's eye caught them). Rendered-identity is then argued by construction (only already-winning rules promoted) AND render-reviewed — not inferred from the byte-diff.
• Hold or exclude volatile build-stamps — a build-date/commit stamp (colophon _build_info) differs between baseline and post builds and falsely flags a byte-diff. Hold it constant or exclude it, and say which (2026-06-10, Stage-G close).
• Two-hat commit separation — name the hat each commit wears: a refactor commit's compiled output is byte-identical (or carries a pre-stated classified delta); a feature commit is where rendered values change. Makes the gate above applicable per-commit instead of per-session (2026-06-08, from Fowler).
• Lossless-relocation gate — restructuring an index or moving prose between files: line-range slices, never retyping, + md5-conservation per moved slice + a link-resolution canary over every ](file.md) and [[wikilink]]. Proven on the MEMORY.md two-file split (2026-07-06) and again on the skill-harvest split (2026-08-01, archive byte-identical over 166,027 bytes).

Mechanical text transforms ("strip/replace token X across the file")

• Token-strip hits prose (2026-06-08) — a whole-file sed s/…!important…// (or any token strip) also removes the token from comments, code-refs, and docstrings, silently mangling prose. Scope the transform to declaration lines, OR compare raw-before vs clean-declaration counts and verify comment integrity after (caught 4 mangled comments via the 56-vs-70 count mismatch; restored). Same family as Count first, then look — the word lives in prose too.

Coverage claims ("all X were handled")

• Exhaustive route/rule census — enumerate ALL routes/rules/sections before classifying any; classify each explicitly; no sampling. (The censused-routes discipline; §I.k.c measuring read.)
• Count first, then look — never read a census through truncation (| head, clipped listings). wc -l before inspection; 3 same-day instances 2026-06-05 (chamber/CM-AI/studium untracked files).
• Re-audit with the TOOL's recognizer, not the classifier that shares its blind spot — when a classifier and the tool it feeds share a predicate, the classifier's mis-classifications masquerade as genuine new categories. Size a gap by running the tool's own recognizer over the corpus.
• A silent safety net is uninformative, not reassuring — a fallback / fail-loud branch / unrecognized kind that never fires across N real cases has not been shown to work; census the substrate for what it should have caught. (Generalized from the negative-grep class to net-silence.)
• Verify at the granularity of the mutation, not the aggregate — a document-wide word-multiset guard PASSES while a per-item swap (a word from note A landing in note B) violates the intent, because the total bag is unchanged. Check per item, keyed by the item's own identity.
• Work-in-omnibus: verify the interior, not the endpoints — when a sidecar brackets one work out of a multi-work source by heading boundaries, sample the span INTERIOR and scan for embedded works and editorial apparatus; endpoint-verification passes over a wrong interior.

Build/render claims ("it works / it looks right")

• Render-and-LOOK — render the artifact and read it before asserting; never reason from the remembered model (caught glued FR colons, feTurbulence errors, viewport misreads).
• Per-type × per-viewport — after any shared CSS/template change, every content type on desktop AND phone (feedback-verify-each-post-type-after-changes).
• Compiled-selector grep gate — after SCSS changes, grep the compiled CSS for retired/duplicated selectors and late overrides (caught opacity duplicates + silent font-family override, 2026-06-03).
• Governed-char byte gate — verify specific codepoints by byte ([8239] probes), not by eye; verify test INPUTS too (the \x202Fc hex-escape trap).
• Measure the box model before theorizing the mechanism — headless-Chrome measurement caught a per-point padding asymmetry where the theory said width (2026-06-09). Reinforced across four days; the standing harness is /measure-render.
• Measure the font's true average advance before a character-count measure — the 0.5em convention misestimated EB Garamond by 32% (0.377em measured); measure over real corpus prose including spaces (2026-06-09).
• Element shot = scrollIntoView({block:'center'}) + full-viewport, never computed clip — box-clip math off getBoundingClientRect kept landing on page-top and burned several shots (2026-06-13).
• CSS mask fills WHITE, not black — a mask/-webkit-mask SVG must fill the shape opaque; a black fill renders BLANK (luminance-vs-alpha). ~6 debug rounds (2026-06-13).
• Live-CSS-patch in _site for in-browser iteration — sed the value in the built CSS and refresh to let the steward judge sizes in the real browser, then lock the chosen value in source SCSS. Turned a multi-round sizing decision into one sitting.

Toolchain claims ("the parser/build does X")

• Measure-toolchain-before-spec — probe actual behavior (ghci/REPL probes) before writing spec text that presumes it («» parse as Str; BulletList misparse; 2026-06-04/05).
• Feature-detect gates can lie — @supports(animation-timeline:view()) returns true on iOS Safari 26 where mobile WebKit will not drive it. A capability probe reports the parser, not the behaviour; verify on the target (2026-06-12).
• Parse-and-validate an artifact before its first machine consumption — a hand-authored YAML/config that humans have only ever read is unvalidated input the moment code consumes it. All four chamber reading-index YAMLs failed on first parse (2026-06-12).

Numeric/accessibility claims

• Compute, don't eyeball — WCAG contrast via calculator (1.84:1 vs an assumed "fine"; 2026-06-03).
• CI-upper-bound + drop-one robustness — STANDARD for every ESCALATE (jurist-ruled 2026-07-04): grade on the one-sided 90% Clopper–Pearson upper bound, never the point estimate, and re-check with any one confirmed case dropped. A point estimate at n=40 conceals how little it constrains.

Remote/persistence claims ("it's backed up / deployed")

• Fresh-clone gate — verify a remote by independent clone and content check, never by push exit-code (chamber-library LFS: the remote held stubs for months while pushes "succeeded"; 2026-06-05).
• Live-fetch gate — verify deploys by fetching the live URL and reading bytes (robots.txt, 403 differentials; 2026-06-05).
• Cross-volume verify-before-delete — rsync -a → exact file-count match + du (never a byte-sum; mine computed 0=0 vacuously) + rc=0 → only then rm the source. Isolate flaky-remote pushes so a hang cannot mask a failure.

Extension claims ("the earlier verification covers this too")

• Re-verify at the extension's scope — the prior pass tested what it tested (feedback-trust-prior-pass-frame; now also a Symmetria §3 flag).
• Extending a tool re-tests its foundations — a widen exercises shared machinery the original's tests never hit, so the extension's --validate must assert the shared invariants too. Building the block-id handler surfaced a latent duplicate-id defect (2026-07-11).
• A stronger gate obliges a retroactive sweep (jurist Q1b) — a stronger check existing and not pointed at what shipped under the weaker one is the exact gap the discipline exists to close.

Causal claims about running systems ("X is what's pinning the CPU / blocking the loop")

• Sampling-profile gate — name the hot path by measurement (process sample / SIGUSR1→CDP CPU profile) before naming the culprit; log shape suggests mechanism but does not prove it (circle-forward read as a retry queue from log census; profile + code read showed fire-and-forget and the pin elsewhere entirely — temporal; 2026-06-06). Corollary: write the inferred-vs-measured split explicitly in any outward claim — when the profile overturns the lean, sharpening costs nothing where retraction would have.
• Exit-code-over-tail gate (2026-06-06): never read success from prettified command output — git push 2>&1 | tail -1 rendered a REJECTION as the soothing fragment "...and the repository exists." Check $? (or let the tool surface stderr whole) before claiming a push/deploy landed. Caught same-session; the byte cost of ; echo exit=$? is zero.
• Verify the RUNNING BINARY's provenance, not source HEAD — check the compiled dist/ mtime and grep the compiled symbols before reasoning about live behaviour. A whole "schema-drift" mechanism inverted on this: HEAD carried a migration the running process did not.
• Quantify the removed cost as the A/B control — when a fix removes a hot operation, time the removed operation on real data rather than racing a flaky end-to-end. (2.56 s vs 0.003 s on the real 676k graph.)
• Confirm a named cause by swap-in — naming the true reference/config/cause is identification, not proof; swap the candidate in and watch the anomaly collapse, ideally to zero.
• Split cause from magnitude before sizing a remedy — a bucket keyed on one summary axis holds heterogeneous causes, and a single label silently implies a single fix at a single scale. Read the substrate and split first.
• Revert-and-redo-smaller — when a gate fails and the cause is not immediately visible, return to the last green state and redo the step smaller; do not debug forward. Named as the reflex we most lack (2026-06-08, from Fowler).

Enforcement rules built on a "never occurs" premise ("ARC English never uses the em dash")

• Corpus-census-the-premise gate (2026-06-07): before an enforcement transform ships, grep the corpus for the premise's counterexamples — the §I.k.b em→en rule's "never uses it" was false for exactly one fixed convention (the Chicago ———. bibliographic repeat, 2 carriers), and it mangled Sources blocks for three days because both the build and the audit verified the rule-as-stated, not the premise-against-the-corpus. One grep would have caught it.

Render-gate comparisons ("the steward rules on render")

• Element-census the comparison page (2026-06-07): before spending the steward's eye, verify the offered page actually carries the element under judgment — the first hr-gate page offered had zero bare <hr> (its dots were an ornament, identical in both builds); the steward saw no difference because none could exist there.
• Pre-registered tool trial (2026-06-07): baseline commit before first contact (every later diff attributes cleanly) + acceptance criteria written into the commit message BEFORE any experience with the tool + diff classification + exercise-before-verdict (drive the real API against questions you genuinely need answered). Made the Basic Memory verdict ~30 min, evidence-grade, attachment-free; the instrument for every future tool trial, BMF milestones included.

Gate-design claims ("the gate would have caught that")

The family earned hardest and most often. A gate's silence is evidence only about what the gate can see.

• The gate itself can be PASS-BUT-FALSELY — a verifier checking an enumerated set of signatures silently passes residue outside the set (verify_conversion reported cruft=0 over 59 raw <sup>). Gate on the class, not the instances.
• Method-class vs calibration — when a metric fails to separate two cases, ask whether it is mis-calibrated or structurally blind. No retuning of a set metric reaches an order-only difference: the blindness is in the method class, not the threshold.
• Coverage never attests order — any permutation preserving local windows passes a coverage comparison, however independent the compared extractors: coverage discards position before comparing. Demonstrated twice on real material and ratified as constitutional text (spec v2.9.0, REVIEWED-84, 2026-07-29).
• A word-guard passes on scrambled text — word presence cannot catch reading-order damage (a two-column read across the gutter): same words, wrong order, PASS. The gate for it is reading-order coherence, not the multiset.
• Positive test at the enforcement path, never a negative grep — for any "can this be bypassed?" property, a grep proves the absence of a string. Drive a known-bad input at the real enforcement path and watch it refuse (2026-07-14, jurist-caught).
• Read the gate's decision code before designing its consumer — reading verify_body_conservation.classify revealed it classifies boundary runs by position+size only and provably cannot confirm class identity; that "no, and knowably no" shaped an honest design and corrected the package mid-build (2026-07-22).
• Implement the relation, not an approximation of it — where code implements a ruled relation or criterion, the acceptance path must BE the relation; a structural approximation (word-aligned matching of a string-defined relation) is a different gate wearing the ruled one's name.
• Seam-probe the artifact — gates test claims, probes test joins — two real defects (empty footnote defs; defs swallowed by a trim) were invisible to every word-level gate because no words were lost, only structure detached. Probe the joins (2026-07-21).
• cmp after apply — a tool's report is not its write decision — strip_cruft --apply prints transform counts before deciding to write; three distinct refusal causes each read as "applied" until cmp proved otherwise (2026-07-21).
• A control must sit at the layer the defect lives in — the PDF-origin classifier passed 8/8 while mis-classifying two real scans: every control exercised the decision rule, so none could test whether the signal set was sufficient. Ask what layer a green check occupies before reading it as coverage (2026-08-01, REVIEWED-83 A1).
• Prose word-guard for faithful structure-cleaning — when a cleaner reflows structure but must preserve prose, gate on a prose-only word-multiset compare (headings, numerals, printed-title runs and markup excluded).

Grounding and citation claims

• Per-claim citation — verify each citation against the specific claim it carries; never let a sub-agent's aggregate "4/4 verified" ride (jurist-caught: a paper cited for a claim it did not support).
• Assert "X is in Y" only after reading Y — grounding must reach the file that HOLDS the fact, not one that describes it. Earned hard 2026-07-17: a claim about test_tools written from the tool's own docstring — a derived self-assessment — was false.
• Byte-check hand classifications at source before citing them — two of my classes and a jurist assessment were overturned by reading the bytes; the correction strengthened the finding (2026-07-19).
• Reconcile against the authoritative source before any "closed/complete/finally" — reconcile against the roadmap or the spec's own named-open items, never the derived close-list.
• "Closable-now" is itself a claim to check — the closable-vs-blocked partition must be reviewed, not asserted; the named-dependency discipline only protects if the closability judgment is checked too (jurist-named).
• Implementation is a second gate — "a gate that passes a document and a build that has to implement it are different tests" (jurist, after their own 07-16 miss surfaced at build orientation). A text passed by reading is re-tested by having to act on it.

Provenance and re-anchor claims

• Re-anchor = re-verify, by sha-match — reconstruct the OLD bound state from git by matching the declared sha, prove the old→live delta is exactly the expected hunks, then re-verify spans. Re-anchoring without re-verifying rebinds a pointer to unexamined content.
• Dry-run-first for bulk file operations — any mass mv/graduate/rename emits the full move-map + collision guard + reference scan (git grep each old basename) before --apply. Proven 3× in one session.
• Re-verify a workflow's per-item dispositions on the real apply — a sub-agent's per-item verdict from a dry-run on a temp copy can differ from the real run; its own audit can disagree with itself.

Governed-document changes

• Governed supersession procedure — cp live→-vNEW → bounded edits, never retyping → diff shows only intended alterations and insertions → header carries Obsoletes + GROUNDED-IN → on promote: freeze old, copy new to live, byte-prove the untouched remainder.
• Reassigned-component check (jurist-minted, REVIEWED-72) — "when a change reassigns a named component, check what else names it." F4 moved the born-digital candidate onto pandoc and silently voided a clause that named the converter.
• Structural safety is PROVISIONAL; end-to-end proof on real material is FINAL — no widen or transform is ruled final on structural reasoning alone. "Same risk as (a)" was true of the matching logic and silent on the rendering logic.

Test-harness claims

• The sandbox must pin the SAME module object the code imports — patch sys.modules, not a freshly-exec'd copy; a _load()-style loader builds a different object and the mispinned sandbox tests nothing.

Gate design — the discrimination gate

• A check must discriminate between two REAL artifacts, one known to have the property and one known to lack it. Same verdict on both = the check has demonstrated nothing, however many synthetic fixtures it passes. Earned 2026-08-02 after four instances in three days of a passing check certifying a property of the code while claiming a property of the result, every one found by a person looking: the vignette field colour bound to a class no element carried · the Fool harness recording degraded:null on a run with no answer · five splitter defects found only by contact with real documents · the §3.3 screen false-passing a package whose Part VII is a collected limitations section.
• The reason positive controls did not catch these: the fixtures were derived from the CHECK, not from the PROPERTY. "What makes this regex fail?" instead of "what makes this claim false?" A control built from the check's own vocabulary inherits its blind spot by construction — the same shape as controls built by extraction leak by construction. Derive fixtures from the property; draw them from real artifacts, since a synthetic negative is written by the same hand as the check.
• Every check states, in its own output, what it did NOT establish — the necessary-but-not-sufficient gap named beside the pass. A check that cannot name its gap does not ship.
• The residue is irreducible and needs a differently-formed reader. Discrimination catches proxy-gaps where a real negative instance exists; it cannot catch a proxy that discriminates on the pair and fails elsewhere. What is left must be looked at before the claim is made, by someone who is not the check's author — Constraint 6 applied to instruments. Reference implementation: dotfiles/claude/governance/fool/test_discrimination.py, which is shown rejecting the §3.3 pattern as it actually shipped.

Estimates and schedules

• Quote a long-job ETA only from an observed rate — twice in one day I gave an ETA from intuition and was wrong by ~30×. Measure rows-per-elapsed on the running job, or benchmark a slice, then quote.

Structure recovery

• One placement engine, two map-producers — chapter-structure recovery is a single engine (insert_chapter_headings: anchor → unique line → replace-block/insert, fail-loud, dry-run) fed by either the EPUB NCX ToC or the OCR page-number method. Anchors are a text-prefix or a literal <span id>.
• Glyph-outline → SVG from a woff2 — fontTools SVGPathPen + BoundsPen over the woff2, y-flipped into SVG space, to build a typographic asset from the real glyphs rather than a trace.

---

Batch-append 2026-08-01 discharging Stroke 2 of the 2026-07-19 skill-harvest FULL REVIEW ("ALL earned ladder entries queued in this register … append to reference-verification-ladder.md with provenance, kin merged in the same pass"). 49 open ladder-kind entries were queued, not the ~25–30 the Stroke estimated; the authorization is by extension ("ALL earned"), not by count. Kin merged into existing claim-classes where one existed; seven new claim-classes added for families that had no home — the largest, gate-design claims, is the one this practice has earned most often. Two queued rows were NOT folded here because Stroke 3 already ruled them skills rather than ladder notes (/measure-render, /clone-test-runtime-fix); they remain build-on-need. Detail and origin for every entry: skill-harvest-archive.md.