58 lines
7.7 KiB
Markdown
58 lines
7.7 KiB
Markdown
---
|
||
name: spec-code-audit
|
||
description: Full bidirectional spec↔code audit — every binding clause traced to read code, every code surface reverse-traced to its governing clause, substantive findings adversarially refuted, gap-map deliverable feeding a promotion/review gate. Use when a spec-governed codebase (ARC/AldineXXI, L1/BMF) needs a confident "is the spec complete and honored?" answer — especially before a jurist read, after heavy spec+code churn, or when an external party asks the completeness question.
|
||
---
|
||
|
||
# Spec↔Code Audit — the four-pass bidirectional method
|
||
|
||
Proven on ARC 2026-06-07 (steward-authorized; `_audits/spec-code-full-audit-2026-06-07.md`, 172 findings, 66-agent fan-out, 48/50 substantive findings surviving adversarial refutation). Generalizes the 05-29 single-spine method (`code-represents-spec-audit-2026-05-29.md`) to the whole corpus, both directions.
|
||
|
||
## When this audit is the right instrument
|
||
|
||
- A reviewer/jurist gate approaches (Stage-G shape: DRAFT→OPERATIVE promotion) and the gap-map is its input document.
|
||
- Heavy churn since the last audit on **both** ends — measure it first: commits since last audit, spec-lines added, code-lines churned (`git log --oneline --since=<last-audit>` + `git diff --stat`). Surface the numbers; the steward decides scope.
|
||
- Someone asks "is the spec complete?" — the honest answer requires the **reverse pass** (code→spec), which single-direction audits never run.
|
||
|
||
NOT this instrument: diff review (/code-review), a single subsystem check (run the relevant pass alone), or a corpus that has no binding spec.
|
||
|
||
## Scope decision (ask the steward)
|
||
|
||
- **Full bidirectional** (the do-once answer): all four passes. ~1 session with fan-out.
|
||
- **Delta-only**: changed territory since the last audit, inheriting prior verdicts — faster, but the answer stays "audited with boundaries", and trust-prior-pass-frame warns the frame inheritance is itself a contamination shape.
|
||
- Always: **do NOT inherit prior audit verdicts.** Unchanged clauses get at least spot re-verification. (On ARC this caught the 05-29 clause-2 verdict being superseded, the §III.b "violation" being remediated, and the "born dormant" transforms being live.)
|
||
|
||
## The four passes
|
||
|
||
**Pass 1 — Spec currency (executor, inline, full read).** Read every spec document end-to-end in this context — no delegation; the inline read is what makes Pass-2 briefs precise and Pass-4 synthesis sound. Hunt: internal contradictions (same doc, two states), cross-spec disagreements (two specs, two lists — find the live arbiter), status honesty (DRAFT labels over live machinery; "draft sketch" preambles on locked corpora), stale implementation-status blocks, stale appendix/index rows, resolved open-questions still open in the text. Number findings (P1.n) and write them into the artifact immediately — the artifact is useful at every stage (non-convergence).
|
||
|
||
**Pass 2 — Spec→code trace (fan-out).** One tracer per spec/section-cluster, briefs authored FROM the Pass-1 read: name the exact clauses, the exact files, and the specific open questions (the P1.n ❓ items). Each clause traced to *read* code AND compiled output.
|
||
|
||
**Pass 3 — Code→spec reverse pass (fan-out; the dimension single-direction audits miss).** One sweeper per surface domain (build program in full ×2 reads · every template incl. orphan-check on non-recursing globs · every style partial vs the selector index, comment-stripped · data/static/assets/scripts). For each surface item: governing clause, or UNGOVERNED. This is where the shipped-behavior-no-spec-records findings live (ARC: recent.html, chrome colours, undocumented routing fields, debug logging, the lone surviving script).
|
||
|
||
**Pass 4 — Synthesis (executor, inline).** Adversarial refutation first (below), then the executor personally re-traces the highest-stakes survivors (batched greps; one command per cluster), then writes the gap-map.
|
||
|
||
## Classification vocabulary (extends 05-29)
|
||
|
||
`✅ match` · `✗ code-gap → [FIX]` · `◐ pre-contract (ships-with-gate)` · `⚠ spec-gap → jurist` · `📜 spec-stale (text overtaken by governed executed change — reconciliation edit, no code change)` · `▷ forward/unimplemented-by-design (check the status block is HONEST about it)` · `⬆ code-ahead-of-spec (shipped ruled behavior the spec doesn't record — the amendment batch)`.
|
||
|
||
Verdict-precision traps the refuters caught on ARC: a field with no consumer cannot be code-ahead-of-spec (it's legacy-residue or spec-gap); a rule whose own status block flags it open is unimplemented-forward, not code-gap; a consolidation rule scoped to a named pair is not violated by unrelated rules elsewhere; "undocumented in spec X" must check the WHOLE corpus before claiming "undocumented anywhere".
|
||
|
||
## Execution pattern — hybrid with adversarial refutation
|
||
|
||
Workflow shape (see the ARC script, reusable): `parallel(tracers) → parallel(sweepers) → refute substantive findings`.
|
||
|
||
- **Findings schema** (force via `schema`): `{clause, verdict(enum), claim, evidence(file:line), confidence, verified-vs-inferred}` + `matches_summary` + `coverage` (exactly what was read; what was NOT traced and why).
|
||
- **Tracer preamble** = Symmetria §5 dispatch preamble + method rules: (1) trace to READ code, never grep-and-assert, cite file:line; (2) counts confirmed TWO WAYS; (3) verify against fresh compiled output, not just source — **check build freshness vs HEAD before launching**; (4) exit codes over prettified output; (5) read-only; (6) the verdict enum with definitions.
|
||
- **Refute** every finding with verdict ∈ {code-gap, spec-gap, code-ahead-of-spec}: one independent refuter each, instructed to re-derive evidence from the files (never trust the finding's citations) and default-refute what doesn't reproduce. Expect ~5% kills + many citation/verdict corrections — the corrections are half the value.
|
||
- **Pipeline defect to guard**: a tracer may pack multiple verdicts into one mega-finding under a single verdict label, so an embedded code-gap escapes refutation (happened on ARC: `$plex-sans-scale` inside a spec-stale blob). Tell tracers ONE claim per finding; scan mega-findings by hand at synthesis.
|
||
- **Executor re-trace**: batch-verify the survivors that will become [FIX]/jurist items (one or two compound bash commands). Record verification depth honestly per item (tracer+refuter vs +executor-spot).
|
||
- 📜/▷ items are textual — the executor re-checks them directly; don't burn refuters on them.
|
||
|
||
## The gap-map deliverable
|
||
|
||
Lives in the repo's audits dir, named `spec-code-full-audit-YYYY-MM-DD.md`. Sections: header (trigger · scope · method · legend) → Pass-1 findings (numbered) → Pass-2/3 execution record + **what conformed cleanly** (the matches are load-bearing for the confidence claim — name them) → the gap-map routed by action: **A `[FIX]`** (table: clause, finding, locus) · **B ships-with-gate** · **C spec-gaps → jurist** · **D code-ahead amendment batch** (ruling shipped → spec edit) · **E spec-stale reconciliation list** (one editing pass) · **F forward/honest** (verify status-block honesty) · **G steward-eye queue** (design decisions surfaced, not resolved) → **gate-readiness verdict** with explicit sequencing (reconciliation pass BEFORE the jurist reads — never ask a jurist to rule against text the work has left behind) → the one-paragraph confident answer to whoever asked.
|
||
|
||
## Costs and calibration (ARC datum)
|
||
|
||
7.9k spec lines + ~1.8k-line build program + 11 templates + 22 partials → 12 tracers + 4 sweepers + 50 refuters = 66 agents, ~4.2M subagent tokens, ~48 min wall. Scale tracer count to spec-section clusters, not files. The multi-agent burn shares the steward's interactive pool — flag before launching; resumable from the workflow journal if a session limit bites.
|