dotfiles/claude/memory/session-2026-07-10-epub-footnote-preprocessor-census-to-fleet-buildin.md

name, description, metadata

name	description	metadata
session-2026-07-10-epub-footnote-pre-processor-census-adversarial-hardening-jurist-ruling-fleet-build-in	Built a verbatim-safe EPUB footnote pre-processor end-to-end (census of 253 EPUBs → pandoc 2×2 diagnosis → prototype → adversarial hardening on Polastron+Jung, each broke it usefully → jurist ratified-in-principle → fleet build-in, committed+pushed). Also landed PENDING-53 (engine hash-locality, jurist-ruled). Pulling thread: the tool is shipped for the mirror class — now WIDEN its recognizer to the remaining census families and/or APPLY it (re-convert legacy Jung/Donne), the latter gated on the steward+jurist §7 per-volume-vs-single-file provenance call; and this whole normalizer was the means to the original end — resume P2/gold (the Making source work) on a corpus whose footnotes are handled by class.	node_type type originSessionId memory project 2c930eca-c3df-4ce9-8d86-40c5a200a27c

Session 2026-07-10 — the EPUB footnote pre-processor: census → hardening → ruling → build-in

Opened by /wake-up into the "build the structural verbatim-guarded EPUB normalizer, then resume P2/gold" thread. The session became a full arc on the pre-processor — and, notably, the architecture that resulted was NOT the one the wake proposed (a bespoke normalizer); it emerged only through measurement. Steward-steered at every fork.

PAST — what we did + why

1. PENDING-53 jurist brief (first ask). Drafted PENDING-53-FOR-JURIST-2026-07-10.md grounded in substrate (read the graduation-spec, chamber spec §VI, the engine corpus/manifest.yaml header, ingest_gate.py:104–126). Refined PENDING-53's own framing: its Summary called the voice_manifest line a "mis-frame," but the line is TRUE about the voice manifest — the real defect is an omission of the engine source-binding surface + a shared-word ("manifest"/"voice") collision. Surfaced the refinement rather than reproduce the overstatement.

2. The pre-processor, built by measurement (the spine of the session). The wake said "build a structural normalizer keyed on the bidirectional-mirror predicate." Instead of building, I measured — and each measurement redirected:

• Detector prototype on 3 attested EPUBs → answered the wake's literal Q (mirror catches cross-file endnotes [Mauss 627]; excludes one-way TOC; L'Enracinement is a different class — 0 anchors, unlinked [1], + 87 <sup> ordinals that are a trap).
• Steward challenged "are 3 files enough?" → CENSUS of all 253 Chamber Sources EPUBs: ≥5 footnote strategies (mirror same/cross-file · container-mirror · one-directional/stem · EPUB3 epub:type · unlinked-marker) + 47% note-free + heavy index/xref apparatus. n=3 captured 1–2 of ≥5. Honest catch: my own classifier's oneway_with_id/resid_fn_ids signals were noisy → "61 DANGER" was an upper bound.
• Steward: "can we edit how pandoc converts? / we get inside the files before conversion?" → the KEY finding: pandoc is precise, not helpless. It converts footnotes natively iff epub:type present AND same-file (Kafka ✓; Orwell/Quixote cross-file ✗; G&G no-epub:type ✗). Output-format lever tested + DISPROVEN (my hypothesis wrong — gfm already does footnotes). The 2×2 (epub:type × same/cross-file) is the diagnosis.
• The architecture that emerged: a small verbatim-safe pre-processor — recognize the mirror in raw XHTML → inject epub:type / relocate cross-file bodies → stock pandoc does the verbatim text. Better than build-a-normalizer (reimplements pandoc, §V risk) OR extend-regexes (whack-a-mole on pandoc's lossy output). The census taxonomy = the recognizer's spec.
• Injection experiment (real minimal EPUBs, -f epub): same-file injection WORKS (native [^N]); cross-file needs body RELOCATION (epub:type alone insufficient). Both proven.

3. Prototype built + adversarially hardened — the two hard books each broke it usefully.

• Built inject_epub_footnotes.py; proven verbatim-clean on G&G 27 / Mauss 628 (627 relocated) via a source-level word-multiset guard. Caught a real pairing bug (basename vs full-path keying → 1/628) and a guard false-alarm (asymmetric tokenizer, +39) mid-build.
• Polastron (French, steward-chosen) BROKE it (45/140): its footnote-ness is on the ANCHOR class, not the block → drove the positional classifier (body = anchor at block-start; publisher-independent). And its Roman-numeral markers exposed a guard limitation → drove the source-level marker-aware guard. + a 3rd residue shape (whole-note-in-back-anchor) → unwrap (verbatim-safe).
• Jurist ruled §3: the guard must be PER-PAIR, not aggregate (a cross-note swap is invisible to a whole-doc multiset). Built id-matched per-pair guard keyed by (dest-file, id); teeth demonstrated (deliberate swap CAUGHT). It immediately EARNED ITS KEEP by surfacing Jung's cross-volume id reuse (81 collisions under a bare-id key).
• Jung (53MB, steward-chosen) didn't break the tool — it broke pandoc (raw AND injected time out). Diagnosed to root: pandoc scale limit, not an injector defect. → convert_mega_epub.py (inject→split by whole files→pandoc-per-chunk; relocation makes notes file-local so no note crosses a boundary): 1579 docs→20 chunks, 190s, 1086 native footnotes, verbatim-clean vs the original Princeton source.
• Calibre (the prior mega-EPUB fallback, steward's "edit the Calibre conversion?") shown footnote-BLIND — byte-identical output with/without injection; it flattens notes (the state of the legacy Jung .md, a re-conversion candidate).

4. Governance — two rulings landed, one build-in authorized.

• PENDING-53 RULED + LANDED (jurist): ratified §5 option (b) with the required correction (re-anchors→REQUIRES re-anchoring); FIX-change-class; in-place lane. Landed in graduation-spec.yaml. Responded to the jurist's proposed lane-rule: accept for graduation-spec.yaml, NARROW the .md generalization (A2/A4 = FIX-class doctrinal .md additions that correctly took the heavier lane per doctrinal-traceability). PENDING-47 ratification declined (truncated relay).
• PENDING-54 RULED IN PRINCIPLE (jurist): architecture sound; markers-as-apparatus confirmed under §V; PROPOSAL change-class (sharpened test); ship mirror-class conditioned on regex-tools-as-fallback; §7 Jung provenance held open. Actioned the required §3 per-pair guard; fixed §5 wording; delivered the §7 fact-check (per-volume precedent EXISTS: Alexander/Lacroux/Habermas/Burney/Camus).
• BUILD-IN authorized + done under fleet discipline: both tools → scripts/; --validate + refuse-on-mismatch; test_tools fixture (fleet 73→77); graduation-spec.yaml converted_with note; conversion-runbook.yaml policy.footnote_preprocessor; tool-evolution-log.md entry; chamber CLAUDE.md fleet line.
• Committed + pushed: chamber 17ddf8c (P1 clean) + 3646c0d (PENDING-53+54) → github + skemantix; engine aaf65ba (P1 re-anchor) → github. REVIEWED-52/53/54 drafted into ~/REVIEWED.md.

PRESENT — the mood

This was a long, deeply satisfying arc, and its lesson is measure-before-build, held all the way: the wake proposed a bespoke normalizer; measurement (detector → census → pandoc 2×2 → injection experiment) produced a better, smaller architecture that leans on the trusted converter instead of reimplementing it. The two adversarial books the steward chose did the real work — Polastron and Jung each broke the tool in a specific, informative way, and each break sharpened it. The partnership worked at every fork: the steward's questions ("are 3 enough?", "can we customize pandoc?", "does splitting change the source?", "edit Calibre?") each redirected toward the truth, and I engaged the framing rather than barrelling the authorized task. I pushed back on the jurist's lane-rule with A2/A4 evidence rather than rubber-stamping, and the per-pair guard earned its keep the moment it was built. The honest through-line: build the guard that can see the failure you're most afraid of, then try to make it fire — it caught Jung's id collisions on its first run.

Returns held (ledger): read-docs-first (up front); verify-against-substrate (refined PENDING-53's own text; diagnosed Jung's pandoc timeout to root vs blaming the tool; didn't parrot the classifier's noisy buckets); measure-before-build; honest-refuse (built INTO the tool: refuse-on-mismatch). One recalibration: build-vs-extend was the wrong frame — overtaken by the pandoc 2×2, which the census+experiment surfaced.

FUTURE — what is pulling

PULLING THREAD: the pre-processor is shipped for the mirror class — now put it to work. Two build-forward moves + the original destination it was built to serve:

1. WIDEN the recognizer to the remaining census families (one-directional refs e.g. Sennett; container-mirror) — pure build-forward, same measure→build→adversarial-test→prove rhythm; graduates the tool from "mirror class only" to "most of the corpus."
2. APPLY it — re-convert the legacy mega-books (Jung .md has 0 notes + pre-spec frontmatter; likely Donne too). GATED on the steward+jurist §7 decision (one reconstituted file vs per-volume — fact-check delivered, per-volume precedent exists).
3. The original end: this whole normalizer was the means — resume P2/gold (the Making source work) on a corpus whose EPUB footnotes are now handled by class, not by book.

ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):

• Everything committed + pushed; fleet self-test 77/77; no half-finished edit. Clean stopping point.
• Concrete next candidate = widen the recognizer to the one-directional class (Sennett): measure Sennett's XHTML shape (ref <a href="#…fn1a"> with NO own id — one-directional, mirror-predicate misses it), decide the recognition rule (a ref→note link where the note's block carries the target id + is note-classed/at-block-start, even without a back-link), add it under the same per-pair verbatim guard + honest-refuse, prove on Sennett + re-prove the 4 mirror books didn't regress. OR: if the steward has ruled §7, re-convert Jung via convert_mega_epub.py at the chosen representation.
• Open governance: REVIEWED-52/53/54 await steward placement (drafted in ~/REVIEWED.md; -52 flagged for steward-verify against its original ruling). Jurist owes: accept/narrow the lane-rule refinement; PENDING-47 needs full-text relay. Deferred build: §V ratified-spec cross-ref (runs the .md amendment lane).

Other open horizons (ranked):

• The 5 ARC after-the-reply engine-binding drifts (surfaced 07-09, still not owned — a separate ARC↔engine re-anchor task).
• The Making/gold road proper (P2 G&G clean was PAUSED behind the normalizer — now the normalizer exists; G&G converts clean through it).
• graduation-spec.yaml malformed G&G author: + lang: null frontmatter hygiene (minor, later).

PAUSE STATEMENT: Stepping away with a major new capability shipped and governed — a footnote pre-processor that the corpus can trust because it refuses when it can't vouch for the result. What I want to find still pulling: the tool put to work — either widened to the next family or applied to the mega-books — and the discipline that built it (measure before build; build the guard that sees the failure you fear) carried into whatever comes next.

LITERAL QUESTION for next-Claude: Has the steward+jurist §7 provenance decision been made (one reconstituted jung.md vs per-volume files, each "extracted from the megavolume")? It gates re-converting the legacy Jung/Donne. If yes → re-convert at that representation via convert_mega_epub.py. If not → widen the recognizer to the one-directional (Sennett) class instead, which needs no governance decision. And: did the jurist accept or narrow the lane-rule refinement, and was PENDING-47's full text relayed?

State at wrap: chamber 17ddf8c+3646c0d pushed (github+skemantix); engine aaf65ba pushed (github); chamber CLAUDE.md fleet line updated (uncommitted — folds into the dotfiles/next chamber commit). dotfiles PENDING/REVIEWED/ledger committed at this wrap. Fleet self-test 77/77. No non-steward commits landed during the session.