session 2026-07-02 evening: Sennett graduation + Family-D footnote tool + corpus standard/method; PENDING-45; audit-map = next pulling thread

2026-07-02 23:22:58 +02:00
parent 0848bf2a55
commit 33de643d2d
4 changed files with 101 additions and 0 deletions
@@ -39,3 +39,30 @@ metadata:
 ## Sub-agent dialogues

 ## Bypasses
+
+## Evening session (wake ~21:22)
+- 2026-07-02T21:22 — Woke into the evening session (~4h after 17:05 wrap). Thread confirmed unchanged: graduate the queue, file each source into Chamber Sources as a step of graduation. Chamber clean @ `7553f59`; nothing moved. Holding drift-pattern *assume-absent-from-filename-grep*. First move if we proceed: one fast EPUB Making source end-to-end through the rail + archive-as-graduation-step proven once.
+
+### Diagnosis (the evening's real finding) — footnote-apparatus is the batch's binding constraint
+- 2026-07-02T~22:30 — Set out to graduate the 7 clean-EPUB Making works (Sebald deferred → Crawford; then steward chose **Sennett reconvert first**). Reconnaissance across all of them surfaced a SYSTEMIC gap, not per-book bad luck:
+  - The graduation tooling's footnote converter (`clean_epub_residue.py --mode footnotes`) covers exactly **3 anchor families**: A=Calibre `nfK`/`anfK`, B=`_r?fn-N`, C=`^([..](#.._chNfnN))`. The actual corpus EPUBs use **≥2 more, uncovered**: Crawford = `filepos` page-anchors (286 markers); **Sennett = `{section}fn{N}` / `{section}fn{N}a`** (e.g. `prologuefn1`/`prologuefn1a`, `ch001fn1`; 536 notes, per-chapter numbered).
+  - **PASS-BUT-FALSELY trap avoided:** `clean_pandoc_html_residue.py`'s `INTERNAL_LINK` unwrap SILENTLY DESTROYS any surviving footnote-link it doesn't recognize (word-guard excludes fn labels, so loss is uncaught). Its abort-guard only fires on the 3 known families → for Sennett/Crawford it previewed "CLEAN" while planning to unwrap → would have reproduced the exact lost-linkage defect. Caught by previewing, not applying.
+  - **Sennett corrected picture:** source EPUB notes are WELL-FORMED (visible `<sup>` in-text markers + verbatim per-chapter-grouped definitions, fully paired by anchor stem). The old canonical's lost linkage was a *calibre* artifact. Reconversion CAN recover it — but ONLY with anchor-aware footnote handling; naive pandoc+existing-cleaners destroys the markers. The 460 `[N](#.._page_N)` "markers" the census first flagged are the INDEX's page-links (drop with Index trim) — a red herring.
+  - Census regexes gave two FALSE family IDs tonight (Crawford mis-read, Sennett `chNfnN=536` false-positive). Recalibration: verify the exact anchor STRING in-file before trusting a family-count grep. (drift: assert-from-grep-hit-without-reading.)
+- **Durable fix identified:** a GENERALIZED anchor-stem footnote converter — pair `[vis](#..._STEM)` ref ↔ `[vis](#..._STEMa)`/`[vis](#..._STEM)` def by the globally-unique STEM (the principle Family-A already uses), emit `[^STEM]`. Do-once; covers A/B/C + Sennett + Crawford + likely the rest. This is the batch's real unblock. Surfaced to steward for a spend-the-evening call vs. a lighter faithful pass.
+- Nothing written to canonical. Sennett canonical file UNTOUCHED. All work in scratch/ (regenerable).
+
+### Family-D footnote handler BUILT + tested (the durable win)
+- 2026-07-02T~23:00 — Steward chose "build the generalized converter." Added **Family D (stem-suffix)** to `clean_epub_residue.py`: ref `[¹](#…_{stem})` ↔ def `[N.](#…_{stem}a)`, keyed by the globally-unique stem (embeds section+number, per-chapter-safe). Tested: **Sennett 294↔294, 0 orphans**; regression-tested **Winnicott (B)** and **Virilio (C)** — caught + fixed a real regression (INLINE_STEM captured C's `r`-prefixed def targets `rchXfnY` as phantom D-refs; fix = also subtract the `r`-prefixed C stem-set in verify_pairing). Conversion is order-safe (D runs after A/B/C consume their anchors); only the verify guard needed the exclusion. Uncommitted; needs tool-evolution-log entry + PENDING (extends PENDING-44's rail).
+- Filepos-positional family (Crawford, 286 markers) explicitly OUT of scope — different pairing model (ref/def don't share a stem), documented as still-owed in the tool docstring.
+
+### Symmetria check — Sennett heading approach (return)
+### CORPUS-QUALITY CENSUS (steward asked: is poor-quality-graduated widespread?) — 2026-07-03 ~00:40
+- Ran both gates across all **332 non-Loeb canonical .md** (+ 952 Loeb DSL excluded). THREE axes:
+  - **Convention/metadata (verify_graduation):** 8 PASS / **324 FAIL**. ~Universal — but almost all failures are frontmatter/title-block (212 missing req fields, 204 no title-block, 109 NO frontmatter at all, 83 canonical≠true). Only 3 fail a text-content check. Mostly = graduated BEFORE the spec+gate existed (2026-07-01). Mechanical class; not alarming per se.
+  - **Text health (verify_conversion):** 222 PASS / **110 FAIL**. ~1/3 have real cruft/OCR/encoding issues — CONCENTRATED in the big OCR'd works: music_performance biographies + Oxford History (Taruskin), the typography/style-guide masters (Bringhurst, Hochuli, Teall, Graves), history tomes (Europe-Davies). This is the genuine quality concern.
+  - **Apparatus integrity: UNMONITORED by either gate.** Only 77/332 have a notes/biblio heading — but Sennett PROVES "has a notes section" ≠ "apparatus intact" (its notes existed but markers were stranded/unlinked; 0 refs). Neither gate checks ref↔def pairing or stranded-apparatus. Scale unknown because nothing measures it. Old-schema signature: 26 files `language:` + 11 `conversion_method:` (Sennett-class).
+- **Verdict: YES widespread, but 3 distinct problems, not one.** Metadata (mechanical, ~universal), text-health (~1/3, OCR'd works), apparatus (unmonitored, ≥1 broken). The known_gap "audit the ~12 already-canonical" UNDERCOUNTS — it's ~110 text-health + an unknown apparatus tail.
+- **Steward directive/instinct: "add apparatus to our audit tools."** AGREED — the unmonitored axis. Proposed apparatus check (verify_graduation extension OR new verify_apparatus): (a) footnote defs `[^x]:` present but unpaired ref `[^x]` → orphan; (b) a Notes/endnotes SECTION present with 0 `[^]` machinery → STRANDED (the calibre-loss signature); (c) ref↔def pairing integrity. Reuse `clean_epub_residue --verify` pairing logic. → NEXT tool-build (not at 1am; deserves its own focused pass). Likely warrants a governed corpus-audit PLAN (the ~110 + apparatus tail).
+
+- 2026-07-02T~23:15 — CHECK on "finish Sennett preserving source heading levels vs full spec-normalization." Named the flag: "preserve levels" is partly convenient (avoids restructuring 129 messy source headings — split `### CHAPTER ONE`+`## Title`, mixed L1–L4 — late at night) AND partly genuinely faithful. Resolution: heading-normalization-vs-faithful-preservation is a CURATORIAL CALL (steward rules those); surface it, don't decide unilaterally. Core defect (lost notes / old frontmatter / `****`) IS fixed by the mechanical pass. Plan: finish approach-INDEPENDENT work (trims + frontmatter + title-block + gate), present near-final to steward WITH the heading question, commit only on their word (avoids do-once rework). Must NOT overclaim "gold bar" — state precisely what was/wasn't normalized.