59 lines
14 KiB
Markdown
59 lines
14 KiB
Markdown
---
|
||
name: session-ledger-2026-07-03
|
||
description: "Practice-of-return ledger maintained by /symmetria — returns, open horizons, recalibrations, authorization moves, sub-agent dialogues, bypasses."
|
||
metadata:
|
||
node_type: memory
|
||
type: feedback
|
||
originSessionId: 5b7ce298-123b-447a-9b86-2442c30bec1f
|
||
---
|
||
|
||
# Session Ledger — 2026-07-03
|
||
|
||
## Returns
|
||
- 2026-07-03 — check before writing `audit_corpus.py`. Three commitments logged: (1) the `verify_graduation.py` refactor touches a GOVERNED gate (PENDING-44) — must be behavior-preserving + PROVEN byte-identical stdout on a sample before trusting; subprocess fallback if proof fails. (2) Scope: conventions+apparatus on the 332 non-Loeb only (Loeb 952 = separate DSL tier; the convention gate would false-FAIL them); text-health on all 1284. (3) Apparatus check CLASSIFIES + emits evidence counts (none/paired-ok/orphaned/stranded-suspect), never adjudicates; validate the classifier against Sennett (→paired-ok) + a real editorial-note file (→not-stranded) before trusting — per census-through-a-pattern flag. Data already surfaced: 8 files with live `[^]:` machinery vs 68 with a Notes heading ⇒ ~60 stranded-suspects.
|
||
|
||
- 2026-07-03 — apparatus classifier blind-spot CAUGHT by validating against real files (not just synthetic self-test): Taruskin/Beethoven had 89/131 `[^x]` refs with ZERO defs (all dangling) yet were classified `notes-editorial` (benign) because an editorial-style heading matched and the `orphaned` branch only fired when defs>0. Fix: refs-present-without-defs → `orphaned`, caught before the heading fallthrough. Without the real-file validation the map would have marked ~dozens of damaged files clean. The census-through-a-pattern flag paid off exactly as predicted.
|
||
- 2026-07-03 — GOVERNED-gate touch (PENDING-44): refactored `verify_graduation.py` to expose `collect_checks()`/`resolve_spec_path()` so the auditor and the gate share one convention source. Proven byte-identical stdout + exit on PASS(Sennett) and FAIL(Plato) samples vs git HEAD before trusting. Behavior-preserving [FIX]; paper note owed alongside PENDING-44/45.
|
||
|
||
## Open horizons
|
||
- 2026-07-03 — MAP BUILT. `audit_corpus.py` + `corpus-quality-ledger.tsv` (1284 rows) + interpreted `corpus-quality-map-2026-07-03.md` with proposed waves. Distribution: Loeb 930/952 clean (corpus healthier in aggregate than the 332-view); non-Loeb 8 clean, 324 A-convention (mechanical), 110 B-textdamage, 70 C-apparatus (NEW axis). Wave A (frontmatter migration, ~99% mechanical, clears 170 A-only to green) recommended as first countdown. AWAITING steward authorization to run a wave — do not begin unilaterally.
|
||
- 2026-07-03T00:00 — Woke into the confirmed pulling thread: build `scripts/audit_corpus.py` + apparatus check → full quality ledger (`_curation/corpus-quality-ledger.tsv`) of ~1,284 works. Verified the map does NOT yet exist. Standing return to hold: **do not resume graduating reactively** — the mole-by-mole reflex is the pattern that produced last session's distress; measure the whole first.
|
||
|
||
- 2026-07-03 — steward challenged the framing (correctly): are the TOOLS spec-compliant before we run any wave? "The tool IS the wave — a stale cleaner run at scale re-corrupts 300 files." Confirmed gaps from reading spec+runbook+tool-log: (1) §VII gate-fix is [OPEN] — verify_conversion heading_density FALSE-FAILS legitimately sectionless works (La Chute), so the map's 59 "B2 structure-recovery" count is CONTAMINATED (conflates damage with sectionless-by-nature). (2) NO gate checks apparatus/footnote-pairing (§V keep_notes) — why 70 damaged files graduated. (3) NO per-language dict-validity check — passed Levi's capacitv/Alanichaean char-substitution. (4) 13/15 mutating/gate tools are STANDALONE (don't reference the governed spec → nothing prevents drift). (5) graduate_to_canonical missing the 2026-07-02 source-archiving wiring (TODO). ⇒ launching fleet compliance audit (3 agents) before authorizing Wave A.
|
||
|
||
- 2026-07-03 — MAP CORRECTION (verified): of 59 heading_density fails, 46 are the §VII sectionless FALSE-FAIL (spec-valid — Plato Republic, Sontag, Berger's About Looking w/25 breaks) and only 13 are genuine structure-loss (some of those, e.g. Rilke Malte, may also be sectionless). B2 was inflated ~4.5×. Running B2 blind would have FORCED headings onto ~46 sectionless works = §II corruption. The gate needs the §VII fix before B2 is even definable. Steward's tools-first instinct vindicated.
|
||
- 2026-07-03 — RETIRE candidates (steward asked: complete kit, nothing useless). Full scripts/ census by ref-count: ref=0 → clean_calibre_artifacts, clear_safe_html_residue, audit_chamber_queryability, promote_fayard_headings; + clean_pdf_running_heads (hardcoded "Auerbach Mimesis" — book-specific); + clean_ocrmac_pagination (superseded by normalize_ocr per runbook). Candidates only — agents confirming redundant-vs-unique before retiring (don't assert-absence-from-grep).
|
||
|
||
## Confidence to recalibrate
|
||
- 2026-07-03 — RECALIBRATION (honest): my morning map-correction "46 sectionless-valid / 13 loss" was WRONG — the quick break-regex counted frontmatter `---` fences AND markdown bullet lists (`^\*\s*\*\s*\*` spanning newlines) as thematic breaks. Fell into census-through-a-pattern on my OWN correction (the Symmetria §3 flag, same day I relied on it for the classifier). TRUE picture: 9 files have real markdown thematic breaks (auto-rescued by §VII fix); ~50 need per-file judgment split into (a) genuinely-sectionless→declare sectionless:true, (b) `• • •`/`**•**`-style separators = a §II.2 break-normalization defect (literal glyph-runs render as prose = no represented structure). The §VII gate fix is CORRECT (asserts "structure represented in markdown"); only my earlier COUNT was contaminated. Lesson re-confirmed: anchor break-detection to the SPEC's break form (`---`), verify the regex against real files, don't span newlines.
|
||
|
||
|
||
## Authorization moves
|
||
|
||
## Sub-agent dialogues
|
||
- 2026-07-03 — Agent C (structure+graduation). audit-agent: PASS (0.95 conf, read every line, greps confirm absence, uncomfortable-not-convenient finding, distinguishes verified/inferred). VERIFIED MYSELF: graduate_to_canonical calls verify_graduation (L63) NOT verify_conversion (0 refs) — only audit_cruft cruft==0 (1 of 5 health checks). git mv no git-add (L133). NO source-archiving (manifest path edits only). graduation-spec.yaml L74-77 DECLARES verify_conversion+verify_graduation+body_word_conservation required → tool violates its own spec. ⇒ the graduation GATE is half-missing: cruft-free-but-damaged files graduate. + verify_graduation itself under-enforces gaps (hardcoded 3 of 6 placeholder patterns). Structure tools healthy (insert_chapter_headings exemplary fail-loud; structure_from_ncx & repair NOT redundant — keep both).
|
||
- 2026-07-03 — Agent A (cleaners). audit-agent: PASS (0.85-0.95, negative-grep rigor, honest empirical-gap caveat). GLOBAL GOOD: NO prophylactic char-normalization in any of 5 (§V Tier-2 honored — the key wave reassurance). strip_cruft (PRIMARY): prose-guard is EXTERNAL/advisory, RESIDUAL>0 prints "don't graduate" then WRITES anyway → port internal word-guard + block-on-residual. clean_pandoc_html_residue: best-guarded, wave-safe. clear_safe_html_residue: orphaned/redundant except <nav> → fold+retire. clean_calibre_artifacts: orphaned + real PASS-BUT-FALSELY (strips <a> footnote links→bare numbers→passes guard; predates keep_notes) → retire or add footnote-preflight. clean_pdf_running_heads: hardwired-Auerbach, no guard/backup, blanket ^\d{1,4}$ strip → NOT wave-safe, rename to oneshot + fix runbook L103.
|
||
|
||
## Meta-findings (the class behind the instances — steward asked "what's hiding in plain sight")
|
||
- 2026-07-03 — (1) ONE failure shape fleet-wide: gates enumerate known-bad SIGNATURES not the INVARIANT (normalize_ocr count-not-identity; verify_conversion cruft-signatures→Taylor <sup>; verify_graduation 3/6 patterns; audit_cruft signature-list; strip_cruft advisory-residual). Wave-0 fixes must all be assert-the-invariant shape. (2) The fleet has NO self-test — every bug found by reading; only audit_corpus (written today) has --validate. ⇒ build a FIXTURE HARNESS (known-good+known-bad per defect class), do each fix test-first. (3) The gate is a discipline not a door — graduate should be the single fail-closed chokepoint. (4) Checks need ONE home shared by gate+map (apparatus lives only in map now → drift); bigger: build_catalogue could carry live gate-status = standing health ledger (§IX coverage). (5) §V Tier-2 conversion-record written only on OCR path, not EPUB/cleaner path — wider §V gap. Ideas 4-ledger + 5-record → follow-on [HARDENING], not Wave 0.
|
||
- 2026-07-03 — STEWARD AUTHORIZED all 8 Wave-0 blockers (after this reflection). Method: fixture harness first, then 8 fixes test-first, invariant-not-signature, gate-as-single-door, shared checks. Tool-fleet-compliance ledger written: _curation/tool-fleet-compliance-2026-07-03.md.
|
||
|
||
## Wave-0 progress (test-first, harness = scripts/test_tools.py)
|
||
- 2026-07-03 — BUILT scripts/test_tools.py (the fleet self-test the tools never had; no pytest dep, matches --validate idiom). Each fix pinned by a fixture asserting the INVARIANT it restored.
|
||
- #5 DONE — verify_conversion §VII sectionless: heading_density passes when 0 headings + real markdown thematic break (---/***) OR sectionless:true; only FLAGS glyph-run separators (• • •) which are a §II.2 break-normalization defect. 3 fixtures green. Recalibrated the map: 9 real-break auto-rescued, ~50 need per-file judgment (not the wrong "46").
|
||
- #1 DONE — graduate_to_canonical.gate_candidate(): the single door now runs BOTH verify_conversion (health) AND verify_graduation (conventions), imported (no more stdout-parsing subprocess). cruft-free-but-damaged + non-conformant both GATED; healthy+conformant passes. 3 fixtures green. Closes the hole that let 110 damaged files graduate.
|
||
- #6 DONE — verify_graduation gap check drives from the FULL spec placeholder_patterns list but only inside a gap FRAME (HTML comment / [bracket]); prose false-positives avoided; STEWARD-RULED exempt. 5 fixtures green; green-set regression still passes.
|
||
- #2 DONE (load-bearing) — normalize_ocr.verbatim_guard now ALWAYS aligns R↔C (merge=concat / doubling=drop-head, per-type counts verified); no more identity-skip when merged_count>0. Catches word-swap-with-balancing-count + unrecorded-join. 4 fixtures green. Signature: (raw,canon,heads,n_merged,n_doubled). Follow-on (noted, not blocker): merge_flag hard-stop policy + wordfreq-venv degradation.
|
||
- #3 DONE — strip_cruft: in-tool prose-safety guard (never bypassable, delegates verify_conversion.prose_delta) + block-on-residual (--allow-residual override); guards run BEFORE backup (refusal = no side effect). 3 fixtures green.
|
||
- #4 DONE — clean_epub_residue: FOOTNOTE_RESIDUAL tripwire raises FootnoteResidual before the TOC-unwrap can silently destroy an uncovered footnote family (#filepos/bare-fn); main aborts loud. Real TOC cross-ref doesn't false-trip. 2 fixtures green.
|
||
- #7 DONE — graduate: git add before git mv (+ plain-move fallback) so untracked batches never silently graduate 0; honest failure count; loud SOURCE-ARCHIVING-NOT-WIRED warning (2026-07-02 directive is a noted follow-on feature-build, not a wave blocker).
|
||
- #8 DONE — retirements: archived 6 orphaned/dead tools → scripts/archive/ (clean_ocrmac_pagination, clean_calibre_artifacts, clear_safe_html_residue, audit_chamber_queryability, promote_fayard_headings, clean_pdf_running_heads[Mimesis one-shot, job done]). FOLDED <nav>-block removal into strip_cruft first (capability preserved, fixture-pinned). Fixed runbook's dangerous generic clean_pdf_running_heads reference. No live tool imports an archived one.
|
||
- ALL 8 WAVE-0 BLOCKERS DONE. Harness scripts/test_tools.py = 21 assertions across 6 tools. Follow-on [HARDENING] (noted, not blockers): source-archiving auto-wire; merge_flag hard-stop + wordfreq-venv pin; §V conversion-record on EPUB/cleaner path; build_catalogue-as-standing-health-ledger (meta-idea 4). NOT committed yet.
|
||
|
||
## Prior-art scans (steward: "incredible we're the only ones"; generative-from-spec principle)
|
||
- 2026-07-03 — Scholarly-encoding scan (audit-agent PASS 0.95, primary sources read). VERDICT: we are NOT first — ~70% of our spec re-derives TEI + CTS + Standard Ebooks. ADOPT: (1) TEI `<choice>` never-destroy-the-original (orig/reg,sic/corr) — vs our replace-and-sidecar; question for gap-3. (2) TEI ch.13 apparatus `<app>/<lem>/<rdg>` — anchor to stable LOCATION not footnote NUMBER + verify resolution → hardens gap-2. (3) CTS URN — structural addressing + EDITION-AS-IDENTITY → what §III should become. (4) Standard Ebooks `se lint` gate (validates our gate arch) + `[Editorial]`-isolated-commit provenance + per-defect detector scripts (audit_corpus template). (5) Distributed Proofreaders: redundant verification AGAINST THE SCAN (single pass can't ground verbatim); HathiTrust: no ground-truth at scale → quality-tiers (validates map+waves). GENUINELY OURS (keep): checksum byte-integrity (TEI/CTS lack it, ~0.75), LLM-literal-citation constraint, corpus-as-trust-substrate governance. TEI ODD/Roma = spec-generates-validator = the steward's generative principle, already a standard → likely the model. 2 scans still out (tooling/spec→validator, bounded-corpora-for-AI).
|
||
|
||
- 2026-07-03 — Tooling/spec→validator scan (audit-agent PASS 0.85, primary sources; the generative-principle answer). TEI ODD/Roma = "One Document Does it All": one spec compiles to BOTH validator AND docs, can't drift = purest generative-from-spec, BUT XML-locked (full adoption = corpus→TEI-XML). VERDICT: realize the principle NOW without full TEI — W3C PROV (PROV-O) = standard vocab for gap-3 conversion-record (adopt, don't invent schema); JSON Schema = free spec→validator for structured layers (.meta.json sidecars, content-types.yml, class enum); PANDOC LUA FILTERS = reimplement cruft/footnote recovery on the parsed AST (structure KNOWN not regex-guessed) → dissolves clean_epub_residue's whole failure family at root; dinglehopper (OCR-D) for GT-diff; ftfy for encoding. GENUINELY OURS (no equivalent anywhere): prose-word-multiset guard (field optimizes plausibility→silent mutation; ours forbids alteration), verbatim-span-in-bounded-source check (RAG does semantic NLI not lexical), checksum integrity. HELD-OPEN FORK [PROPOSAL]: markdown+JSON-Schema now (proportionate) vs TEI-XML+ODD if engine fidelity outgrows markdown — name, don't resolve. 1 scan out (bounded-corpora-for-AI). Then: consolidated synthesis → jurist spec-revision brief (PENDING-46).
|
||
|
||
## Bypasses
|