7.7 KiB
name, description
| name | description |
|---|---|
| spec-code-audit | Full bidirectional spec↔code audit — every binding clause traced to read code, every code surface reverse-traced to its governing clause, substantive findings adversarially refuted, gap-map deliverable feeding a promotion/review gate. Use when a spec-governed codebase (ARC/AldineXXI, L1/BMF) needs a confident "is the spec complete and honored?" answer — especially before a jurist read, after heavy spec+code churn, or when an external party asks the completeness question. |
Spec↔Code Audit — the four-pass bidirectional method
Proven on ARC 2026-06-07 (steward-authorized; _audits/spec-code-full-audit-2026-06-07.md, 172 findings, 66-agent fan-out, 48/50 substantive findings surviving adversarial refutation). Generalizes the 05-29 single-spine method (code-represents-spec-audit-2026-05-29.md) to the whole corpus, both directions.
When this audit is the right instrument
- A reviewer/jurist gate approaches (Stage-G shape: DRAFT→OPERATIVE promotion) and the gap-map is its input document.
- Heavy churn since the last audit on both ends — measure it first: commits since last audit, spec-lines added, code-lines churned (
git log --oneline --since=<last-audit>+git diff --stat). Surface the numbers; the steward decides scope. - Someone asks "is the spec complete?" — the honest answer requires the reverse pass (code→spec), which single-direction audits never run.
NOT this instrument: diff review (/code-review), a single subsystem check (run the relevant pass alone), or a corpus that has no binding spec.
Scope decision (ask the steward)
- Full bidirectional (the do-once answer): all four passes. ~1 session with fan-out.
- Delta-only: changed territory since the last audit, inheriting prior verdicts — faster, but the answer stays "audited with boundaries", and trust-prior-pass-frame warns the frame inheritance is itself a contamination shape.
- Always: do NOT inherit prior audit verdicts. Unchanged clauses get at least spot re-verification. (On ARC this caught the 05-29 clause-2 verdict being superseded, the §III.b "violation" being remediated, and the "born dormant" transforms being live.)
The four passes
Pass 1 — Spec currency (executor, inline, full read). Read every spec document end-to-end in this context — no delegation; the inline read is what makes Pass-2 briefs precise and Pass-4 synthesis sound. Hunt: internal contradictions (same doc, two states), cross-spec disagreements (two specs, two lists — find the live arbiter), status honesty (DRAFT labels over live machinery; "draft sketch" preambles on locked corpora), stale implementation-status blocks, stale appendix/index rows, resolved open-questions still open in the text. Number findings (P1.n) and write them into the artifact immediately — the artifact is useful at every stage (non-convergence).
Pass 2 — Spec→code trace (fan-out). One tracer per spec/section-cluster, briefs authored FROM the Pass-1 read: name the exact clauses, the exact files, and the specific open questions (the P1.n ❓ items). Each clause traced to read code AND compiled output.
Pass 3 — Code→spec reverse pass (fan-out; the dimension single-direction audits miss). One sweeper per surface domain (build program in full ×2 reads · every template incl. orphan-check on non-recursing globs · every style partial vs the selector index, comment-stripped · data/static/assets/scripts). For each surface item: governing clause, or UNGOVERNED. This is where the shipped-behavior-no-spec-records findings live (ARC: recent.html, chrome colours, undocumented routing fields, debug logging, the lone surviving script).
Pass 4 — Synthesis (executor, inline). Adversarial refutation first (below), then the executor personally re-traces the highest-stakes survivors (batched greps; one command per cluster), then writes the gap-map.
Classification vocabulary (extends 05-29)
✅ match · ✗ code-gap → [FIX] · ◐ pre-contract (ships-with-gate) · ⚠ spec-gap → jurist · 📜 spec-stale (text overtaken by governed executed change — reconciliation edit, no code change) · ▷ forward/unimplemented-by-design (check the status block is HONEST about it) · ⬆ code-ahead-of-spec (shipped ruled behavior the spec doesn't record — the amendment batch).
Verdict-precision traps the refuters caught on ARC: a field with no consumer cannot be code-ahead-of-spec (it's legacy-residue or spec-gap); a rule whose own status block flags it open is unimplemented-forward, not code-gap; a consolidation rule scoped to a named pair is not violated by unrelated rules elsewhere; "undocumented in spec X" must check the WHOLE corpus before claiming "undocumented anywhere".
Execution pattern — hybrid with adversarial refutation
Workflow shape (see the ARC script, reusable): parallel(tracers) → parallel(sweepers) → refute substantive findings.
- Findings schema (force via
schema):{clause, verdict(enum), claim, evidence(file:line), confidence, verified-vs-inferred}+matches_summary+coverage(exactly what was read; what was NOT traced and why). - Tracer preamble = Symmetria §5 dispatch preamble + method rules: (1) trace to READ code, never grep-and-assert, cite file:line; (2) counts confirmed TWO WAYS; (3) verify against fresh compiled output, not just source — check build freshness vs HEAD before launching; (4) exit codes over prettified output; (5) read-only; (6) the verdict enum with definitions.
- Refute every finding with verdict ∈ {code-gap, spec-gap, code-ahead-of-spec}: one independent refuter each, instructed to re-derive evidence from the files (never trust the finding's citations) and default-refute what doesn't reproduce. Expect ~5% kills + many citation/verdict corrections — the corrections are half the value.
- Pipeline defect to guard: a tracer may pack multiple verdicts into one mega-finding under a single verdict label, so an embedded code-gap escapes refutation (happened on ARC:
$plex-sans-scaleinside a spec-stale blob). Tell tracers ONE claim per finding; scan mega-findings by hand at synthesis. - Executor re-trace: batch-verify the survivors that will become [FIX]/jurist items (one or two compound bash commands). Record verification depth honestly per item (tracer+refuter vs +executor-spot).
- 📜/▷ items are textual — the executor re-checks them directly; don't burn refuters on them.
The gap-map deliverable
Lives in the repo's audits dir, named spec-code-full-audit-YYYY-MM-DD.md. Sections: header (trigger · scope · method · legend) → Pass-1 findings (numbered) → Pass-2/3 execution record + what conformed cleanly (the matches are load-bearing for the confidence claim — name them) → the gap-map routed by action: A [FIX] (table: clause, finding, locus) · B ships-with-gate · C spec-gaps → jurist · D code-ahead amendment batch (ruling shipped → spec edit) · E spec-stale reconciliation list (one editing pass) · F forward/honest (verify status-block honesty) · G steward-eye queue (design decisions surfaced, not resolved) → gate-readiness verdict with explicit sequencing (reconciliation pass BEFORE the jurist reads — never ask a jurist to rule against text the work has left behind) → the one-paragraph confident answer to whoever asked.
Costs and calibration (ARC datum)
7.9k spec lines + ~1.8k-line build program + 11 templates + 22 partials → 12 tracers + 4 sweepers + 50 refuters = 66 agents, ~4.2M subagent tokens, ~48 min wall. Scale tracer count to spec-section clusters, not files. The multi-agent burn shares the steward's interactive pool — flag before launching; resumable from the workflow journal if a session limit bites.