dotfiles/claude/memory/session-2026-07-02-evening-corpus-standard-and-the-method.md

name, description, metadata

name	description	metadata
Session 2026-07-02 evening — Sennett + Family-D tool + the corpus standard & the method	Reconverted+graduated Sennett (built/tested/committed the Family-D footnote handler, 10cff59). A corpus-quality census then revealed 'graduated ≠ verified' is corpus-wide (8/332 pass conventions, 110/332 text-damaged, apparatus unmonitored). Steward named the vision slipping ('sand through fingers', 'whack-a-mole') and set the STANDARD (verified verbatim fidelity + provenance + apparatus for anything the engine cites) and the METHOD (measure-the-whole-first, then remediate-by-defect-class in finite waves). PULLING THREAD: build audit_corpus.py + the apparatus check → produce the full quality MAP of all 1,284 works (the instrument that ends the whack-a-mole), then plan waves from data.	type originSessionId project 019pnD8uNFC9Q2dbLQa3MBDc

Session 2026-07-02 evening — Sennett, the Family-D tool, and the way through

Started as a short evening session to graduate a few Making-sequence EPUBs "in that order, see how many we can do." Became something more important: a durable tool, a corpus-wide diagnosis, and — most of all — a method and a standard the steward now wants to build against. Only ONE work graduated; that is not the session's value.

PAST — what we did + why

Graduation-order plan (Sebald → Crawford → Winnicott → Agamben → Steiner → Calasso → Celan). Recon killed the "clean fast-path" framing: each has a footnote-family issue OR an image question.

• Sebald DEFERRED (steward call): integral uncaptioned photographs = content, not decoration; the chamber is a text corpus. Which work (Rings of Saturn / Austerlitz) also unresolved. Needs an image policy.
• Celan flagged: verse + bilingual, not a prose fast-path.
• Image census: mostly decoration (covers, ornaments ×18–53) + a few possible-content figures per book (Winnicott squiggle, Calasso root2) — per-book triage owed.

Footnote-family gap = the batch's binding constraint (the real diagnosis). The rail's converter (clean_epub_residue.py) covered 3 anchor families (A Calibre nfK, B _fn-N, C caret chNfnN). The corpus uses ≥2 more, UNCOVERED: Crawford = filepos page-anchors (286), Sennett = {section}fn{N}/{…}a (294). PASS-BUT-FALSELY trap avoided: clean_pandoc_html_residue's INTERNAL_LINK unwrap silently DESTROYS any footnote-link it doesn't recognize (word-guard excludes fn labels) → for Sennett/Crawford it previewed "CLEAN" while about to destroy the markers = reproduce the exact lost-linkage defect. Caught by previewing, not applying. (Census regexes gave 2 FALSE family IDs — recalibration: read the exact anchor STRING before trusting a family-count grep.)

BUILT Family-D (stem-suffix) footnote handler (steward-authorized "build the generalized converter"). Ref [¹](#…_{stem}) ↔ def [N.](#…_{stem}a), keyed by the globally-unique stem (section+number, per-chapter-safe). Conversion runs AFTER A/B/C (their anchors consumed first). Tested + regression-tested: Sennett 294↔294 (0 orphans); Winnicott (B) clean; Virilio (C) — CAUGHT+FIXED a real regression (INLINE_STEM captured C's r-prefixed def targets rchXfnY as phantom D-refs → 72 false orphans → verify would wrongly ABORT; fix = subtract both C stem-set AND its r-prefixed form in verify_pairing; conversion was always order-safe). Lesson logged: a broad regex added to a multi-family converter must be tested against EVERY family's fixtures. Crawford's filepos = a DIFFERENT pairing model (ref/def don't share a stem) — OUT of scope, documented as still-owed in the docstring.

Reconverted + graduated Sennett The Craftsman (steward chose "Sennett first" — its old canonical was calibre-era: lost note-linkage, old-schema frontmatter, 27 **** drop-cap heading residue). New pipeline: pandoc → Family-D footnotes (294 recovered+paired) → residue strip (decoration images dropped) → trim (front boilerplate/dedication/ToC/Index) → frontmatter (current schema) + title block + epigraph ("travail, opium unique") → full spec heading-normalization (steward chose "normalize first": merged 10 split ### CHAPTER N+## Title → ## N. Title, title-cased PART labels, promoted book-divisions to L1, recovered the Conclusion which was a SETEXT heading rendered as plain text, region-demoted subsections). Both gates PASS; body prose word-guarded (delta fully explained: stranded note-digits → [^] labels + frontmatter + dropped index/toc — NO content loss, verified by multiset comparison). Source already in Chamber Sources (sha c46fc577), not engine-registered. Committed 10cff59 (chamber-library, local, ahead 1). tool-evolution-log + PENDING-45 written.

CORPUS-QUALITY CENSUS (steward asked: is poor-quality-graduated widespread?). Ran both gates across all 332 non-Loeb canonical .md (+952 Loeb excluded):

• Conventions (verify_graduation): 8 PASS / 324 FAIL — but ~all frontmatter/title-block (pre-spec; the gate is 2026-07-01). Mechanical class.
• Text health (verify_conversion): 222 PASS / 110 FAIL — real OCR/cruft damage, CONCENTRATED in big scanned works (Bringhurst, Hochuli, Taruskin, music biographies, Europe-Davies). The genuine concern.
• Apparatus: UNMONITORED by either gate. Only 77/332 have a notes heading; Sennett proves "has notes" ≠ "apparatus intact." Old-schema signature: 26 language: + 11 conversion_method:.

PRESENT — mood / the pivot

This session's weight was NOT the graduation. Midway, the steward felt the recursion — "this is starting to feel like a mandelbrot set", then "whack-a-mole… the vision slipping… clutching sand… I need purchase… this was supposed to be the part we could solve; if this is so difficult, I tremble at the thought of making the engine work." Real distress about the whole vision.

Two things answered it:

1. The standard (steward asked directly, "honest answer even if I won't like it"): the corpus must aspire to verified verbatim fidelity + correct provenance (attribution/structure/apparatus) for EVERYTHING the engine may cite — a floor, not a tier. Because the engine can't tell corrupt from clean; it cites both with equal confidence; confidence-without-correctness is the exact contamination the project exists to defeat. A quietly-corrupt bounded corpus is WORSE than none (the L1 line, applied). "Graduated" has meant converted+filed, not verified faithful — a weaker claim than the engine needs. Escape from despair: you don't need 1,284 perfect works — you need a VERIFIED, FENCED subset + a boundary that never lies about its edge (boundedness=trust, turned on the library). Fence the engine to green NOW; grow green to 100% by waves.
2. The method (steward: "how do we systematically work through this? I need the WHOLE thing, queryable, reliable"): whack-a-mole = working REACTIVELY, per-file, unbounded. The cure = invert the order: MEASURE the whole first (one auditor → one quality map), then REMEDIATE BY DEFECT-CLASS in finite waves, each a countdown. You never get surprised mid-file again because the map told you everything up front.

Returns/recalibrations (from ledger): owned working mole-by-mole tonight (the pattern to break = build the map first); avoided the PASS-BUT-FALSELY footnote-destruction by previewing; caught own false family-ID greps (read the exact string, not the count); named the "convenient-vs-faithful" heading choice and surfaced it rather than deciding unilaterally.

FUTURE — what is pulling

PULLING THREAD (singular): Build the instrument and draw the map — audit_corpus.py + the apparatus check → a full machine-readable quality ledger of all 1,284 works, one row per file (conventions / text-health / apparatus / provenance-link + defect-class + source-on-hand). This is the deliverable that ends the whack-a-mole; from it, remediation waves are planned from data. The next session's job is NOT "graduate more works" — it is the MAP.

ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):

• chamber-library main @ 10cff59 (ahead 1; pushed at wrap or note). Family-D tool live in scripts/clean_epub_residue.py.
• First move: build scripts/audit_corpus.py that runs, per canonical file: verify_graduation + verify_conversion + a NEW apparatus check + provenance-link check → writes _curation/corpus-quality-ledger.tsv (or .json). Most checks exist; assemble them. The apparatus check (design in ledger + PENDING context): (a) footnote defs [^x]: present but unpaired ref [^x] → orphan; (b) a Notes/endnotes SECTION present with 0 [^] machinery → STRANDED (the calibre-loss signature); (c) ref↔def pairing (reuse clean_epub_residue --verify logic).
• Then: classify the ledger into defect-classes and present the wave plan. Expected classes: A convention-only (majority, batchable via a frontmatter-migration script — Wave A alone ≈ 8→200 pass); B text-damage (~110, reconvert from archive, parallelizable); C apparatus (stranded notes, count TBD, reconvert like Sennett). Overlaps exist (Sennett was A+C).

Other open horizons, ranked:

• The remediation waves themselves (after the map): A (frontmatter migration, near-automatic, biggest count-mover) → B (reconversion of OCR-damaged, finite list) → C (apparatus). Each a countdown; fence the engine to the green set as it grows.
• Sebald image policy (deferred; blocks Sebald + informs all image-bearing works).
• Crawford filepos footnote handler (the 2nd uncovered family; different pairing model, still owed).
• The original Making batch (Winnicott/Agamben/Steiner/Calasso/Celan) — now SUBORDINATE to the map/waves; don't resume graduating reactively.
• Paper debt: REVIEWED-44/45/46 unplaced; PENDING-45 filed tonight.

PAUSE STATEMENT: Away now, at a low-morale but clarified moment. The vision felt like slipping sand tonight; the answer given was a method that turns the infinite fractal into a finite, countable, wave-by-wave siege. What I most want to find still pulling on return: the map — that the next hand builds the auditor and produces the quality ledger of the whole corpus, so we stop meeting the edge one grain at a time and start seeing the whole boundary at once. Do NOT resume graduating works reactively; build the instrument first.

LITERAL QUESTION for next-Claude: Did the map get built — does audit_corpus.py exist and has it produced a full quality ledger of all 1,284 works? If so, what did the real defect-class distribution turn out to be (vs tonight's 332-file estimate), and is Wave A (frontmatter migration) the right first countdown to run?

State: chamber-library main @ 10cff59 (Family-D tool + Sennett graduation committed; 2 pre-existing untracked leftovers _scratch/, nature-of-order-alexander.md NOT mine). Sennett canonical replaced + verified. PENDING-45 filed. Census numbers captured in the ledger. Nothing else touched canonical.