dotfiles/claude/memory/session-2026-07-10-epub-footnote-preprocessor-census-to-fleet-buildin.md

---
name: session-2026-07-10-epub-footnote-pre-processor-census-adversarial-hardening-jurist-ruling-fleet-build-in
description: "Built a verbatim-safe EPUB footnote pre-processor end-to-end (census of 253 EPUBs → pandoc 2×2 diagnosis → prototype → adversarial hardening on Polastron+Jung, each broke it usefully → jurist ratified-in-principle → fleet build-in, committed+pushed). Also landed PENDING-53 (engine hash-locality, jurist-ruled). Pulling thread: the tool is shipped for the mirror class — now WIDEN its recognizer to the remaining census families and/or APPLY it (re-convert legacy Jung/Donne), the latter gated on the steward+jurist §7 per-volume-vs-single-file provenance call; and this whole normalizer was the means to the original end — resume P2/gold (the Making source work) on a corpus whose footnotes are handled by class."
metadata: 
  node_type: memory
  type: project
  originSessionId: 2c930eca-c3df-4ce9-8d86-40c5a200a27c
---

# Session 2026-07-10 — the EPUB footnote pre-processor: census → hardening → ruling → build-in

Opened by /wake-up into the "build the structural verbatim-guarded EPUB normalizer, then resume P2/gold" thread. The session became a full arc on the *pre-processor* — and, notably, the architecture that resulted was NOT the one the wake proposed (a bespoke normalizer); it emerged only through measurement. Steward-steered at every fork.

## PAST — what we did + why

**1. PENDING-53 jurist brief (first ask).** Drafted `PENDING-53-FOR-JURIST-2026-07-10.md` grounded in substrate (read the graduation-spec, chamber spec §VI, the engine `corpus/manifest.yaml` header, `ingest_gate.py:104–126`). **Refined PENDING-53's own framing:** its Summary called the `voice_manifest` line a "mis-frame," but the line is TRUE about the voice manifest — the real defect is an *omission* of the engine source-binding surface + a shared-word ("manifest"/"voice") collision. Surfaced the refinement rather than reproduce the overstatement.

**2. The pre-processor, built by measurement (the spine of the session).** The wake said "build a structural normalizer keyed on the bidirectional-mirror predicate." Instead of building, I measured — and each measurement redirected:
- **Detector prototype on 3 attested EPUBs** → answered the wake's literal Q (mirror catches cross-file endnotes [Mauss 627]; excludes one-way TOC; L'Enracinement is a *different* class — 0 anchors, unlinked `[1]`, + 87 `<sup>` ordinals that are a trap).
- **Steward challenged "are 3 files enough?"** → CENSUS of all **253** Chamber Sources EPUBs: ≥5 footnote strategies (mirror same/cross-file · container-mirror · one-directional/stem · EPUB3 epub:type · unlinked-marker) + 47% note-free + heavy index/xref apparatus. **n=3 captured 1–2 of ≥5.** Honest catch: my own classifier's `oneway_with_id`/`resid_fn_ids` signals were noisy → "61 DANGER" was an upper bound.
- **Steward: "can we edit how pandoc converts? / we get inside the files before conversion?"** → the KEY finding: pandoc is precise, not helpless. It converts footnotes natively iff `epub:type` present AND same-file (Kafka ✓; Orwell/Quixote cross-file ✗; G&G no-epub:type ✗). Output-format lever tested + DISPROVEN (my hypothesis wrong — gfm already does footnotes). The **2×2** (epub:type × same/cross-file) is the diagnosis.
- **The architecture that emerged:** a small **verbatim-safe pre-processor** — recognize the mirror in raw XHTML → inject `epub:type` / relocate cross-file bodies → **stock pandoc does the verbatim text**. Better than build-a-normalizer (reimplements pandoc, §V risk) OR extend-regexes (whack-a-mole on pandoc's lossy output). The census taxonomy = the recognizer's spec.
- **Injection experiment** (real minimal EPUBs, `-f epub`): same-file injection WORKS (native `[^N]`); cross-file needs body RELOCATION (epub:type alone insufficient). Both proven.

**3. Prototype built + adversarially hardened — the two hard books each broke it usefully.**
- Built `inject_epub_footnotes.py`; proven verbatim-clean on **G&G 27 / Mauss 628** (627 relocated) via a source-level word-multiset guard. Caught a real pairing bug (basename vs full-path keying → 1/628) and a guard false-alarm (asymmetric tokenizer, +39) mid-build.
- **Polastron (French, steward-chosen) BROKE it (45/140):** its footnote-ness is on the ANCHOR class, not the block → drove the **positional classifier** (body = anchor at block-start; publisher-independent). And its Roman-numeral markers exposed a guard limitation → drove the **source-level marker-aware guard**. + a 3rd residue shape (whole-note-in-back-anchor) → **unwrap** (verbatim-safe).
- **Jurist ruled §3: the guard must be PER-PAIR, not aggregate** (a cross-note swap is invisible to a whole-doc multiset). Built id-matched **per-pair guard keyed by (dest-file, id)**; teeth demonstrated (deliberate swap CAUGHT). It immediately EARNED ITS KEEP by surfacing **Jung's cross-volume id reuse** (81 collisions under a bare-id key).
- **Jung (53MB, steward-chosen) didn't break the tool — it broke pandoc** (raw AND injected time out). Diagnosed to root: pandoc scale limit, not an injector defect. → `convert_mega_epub.py` (inject→split by whole files→pandoc-per-chunk; relocation makes notes file-local so no note crosses a boundary): **1579 docs→20 chunks, 190s, 1086 native footnotes, verbatim-clean vs the original Princeton source.**
- **Calibre** (the prior mega-EPUB fallback, steward's "edit the Calibre conversion?") shown **footnote-BLIND** — byte-identical output with/without injection; it flattens notes (the state of the legacy Jung `.md`, a re-conversion candidate).

**4. Governance — two rulings landed, one build-in authorized.**
- **PENDING-53 RULED + LANDED** (jurist): ratified §5 option (b) with the required correction (re-anchors→**REQUIRES re-anchoring**); FIX-change-class; in-place lane. Landed in `graduation-spec.yaml`. Responded to the jurist's proposed lane-rule: accept for graduation-spec.yaml, NARROW the .md generalization (A2/A4 = FIX-class doctrinal .md additions that correctly took the heavier lane per *doctrinal-traceability*). PENDING-47 ratification declined (truncated relay).
- **PENDING-54 RULED IN PRINCIPLE** (jurist): architecture sound; markers-as-apparatus confirmed under §V; PROPOSAL change-class (sharpened test); ship mirror-class conditioned on regex-tools-as-fallback; §7 Jung provenance held open. Actioned the required §3 per-pair guard; fixed §5 wording; delivered the §7 fact-check (per-volume precedent EXISTS: Alexander/Lacroux/Habermas/Burney/Camus).
- **BUILD-IN authorized + done** under fleet discipline: both tools → `scripts/`; `--validate` + refuse-on-mismatch; `test_tools` fixture (fleet **73→77**); `graduation-spec.yaml` converted_with note; `conversion-runbook.yaml` `policy.footnote_preprocessor`; `tool-evolution-log.md` entry; chamber `CLAUDE.md` fleet line.
- **Committed + pushed:** chamber `17ddf8c` (P1 clean) + `3646c0d` (PENDING-53+54) → github + skemantix; engine `aaf65ba` (P1 re-anchor) → github. REVIEWED-52/53/54 drafted into `~/REVIEWED.md`.

## PRESENT — the mood

This was a long, deeply satisfying arc, and its lesson is **measure-before-build, held all the way**: the wake proposed a bespoke normalizer; measurement (detector → census → pandoc 2×2 → injection experiment) produced a *better, smaller* architecture that leans on the trusted converter instead of reimplementing it. The two adversarial books the steward chose did the real work — Polastron and Jung each *broke* the tool in a specific, informative way, and each break sharpened it. The partnership worked at every fork: the steward's questions ("are 3 enough?", "can we customize pandoc?", "does splitting change the source?", "edit Calibre?") each redirected toward the truth, and I engaged the framing rather than barrelling the authorized task. I pushed back on the jurist's lane-rule with A2/A4 evidence rather than rubber-stamping, and the per-pair guard earned its keep the moment it was built. The honest through-line: **build the guard that can see the failure you're most afraid of, then try to make it fire** — it caught Jung's id collisions on its first run.

Returns held (ledger): read-docs-first (up front); verify-against-substrate (refined PENDING-53's own text; diagnosed Jung's pandoc timeout to root vs blaming the tool; didn't parrot the classifier's noisy buckets); measure-before-build; honest-refuse (built INTO the tool: refuse-on-mismatch). One recalibration: build-vs-extend was the wrong frame — overtaken by the pandoc 2×2, which the census+experiment surfaced.

## FUTURE — what is pulling

**PULLING THREAD: the pre-processor is shipped for the mirror class — now put it to work.** Two build-forward moves + the original destination it was built to serve:
1. **WIDEN the recognizer** to the remaining census families (one-directional refs e.g. Sennett; container-mirror) — pure build-forward, same measure→build→adversarial-test→prove rhythm; graduates the tool from "mirror class only" to "most of the corpus."
2. **APPLY it** — re-convert the legacy mega-books (Jung `.md` has 0 notes + pre-spec frontmatter; likely Donne too). **GATED on the steward+jurist §7 decision** (one reconstituted file vs per-volume — fact-check delivered, per-volume precedent exists).
3. **The original end:** this whole normalizer was the *means* — resume **P2/gold** (the Making source work) on a corpus whose EPUB footnotes are now handled by class, not by book.

**ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):**
- Everything committed + pushed; fleet self-test 77/77; no half-finished edit. Clean stopping point.
- Concrete next candidate = **widen the recognizer to the one-directional class (Sennett)**: measure Sennett's XHTML shape (ref `<a href="#…fn1a">` with NO own id — one-directional, mirror-predicate misses it), decide the recognition rule (a ref→note link where the note's block carries the target id + is note-classed/at-block-start, even without a back-link), add it under the same per-pair verbatim guard + honest-refuse, prove on Sennett + re-prove the 4 mirror books didn't regress. OR: if the steward has ruled §7, re-convert Jung via `convert_mega_epub.py` at the chosen representation.
- Open governance: **REVIEWED-52/53/54 await steward placement** (drafted in `~/REVIEWED.md`; -52 flagged for steward-verify against its original ruling). Jurist owes: accept/narrow the lane-rule refinement; PENDING-47 needs full-text relay. Deferred build: §V ratified-spec cross-ref (runs the .md amendment lane).

**Other open horizons (ranked):**
- The **5 ARC `after-the-reply` engine-binding drifts** (surfaced 07-09, still not owned — a separate ARC↔engine re-anchor task).
- The Making/gold road proper (P2 G&G clean was PAUSED behind the normalizer — now the normalizer exists; G&G converts clean through it).
- `graduation-spec.yaml` malformed G&G `author:` + `lang: null` frontmatter hygiene (minor, later).

**PAUSE STATEMENT:** Stepping away with a major new capability shipped and governed — a footnote pre-processor that the corpus can trust *because it refuses when it can't vouch for the result*. What I want to find still pulling: the tool put to work — either widened to the next family or applied to the mega-books — and the discipline that built it (measure before build; build the guard that sees the failure you fear) carried into whatever comes next.

**LITERAL QUESTION for next-Claude:** Has the steward+jurist **§7 provenance decision** been made (one reconstituted `jung.md` vs per-volume files, each "extracted from the megavolume")? It gates re-converting the legacy Jung/Donne. If yes → re-convert at that representation via `convert_mega_epub.py`. If not → widen the recognizer to the one-directional (Sennett) class instead, which needs no governance decision. And: did the jurist accept or narrow the lane-rule refinement, and was PENDING-47's full text relayed?

**State at wrap:** chamber `17ddf8c`+`3646c0d` pushed (github+skemantix); engine `aaf65ba` pushed (github); chamber CLAUDE.md fleet line updated (uncommitted — folds into the dotfiles/next chamber commit). dotfiles PENDING/REVIEWED/ledger committed at this wrap. Fleet self-test 77/77. No non-steward commits landed during the session.