62 lines
11 KiB
Markdown
62 lines
11 KiB
Markdown
---
|
||
name: Session 2026-07-02 evening — Sennett + Family-D tool + the corpus standard & the method
|
||
description: "Reconverted+graduated Sennett (built/tested/committed the Family-D footnote handler, 10cff59). A corpus-quality census then revealed 'graduated ≠ verified' is corpus-wide (8/332 pass conventions, 110/332 text-damaged, apparatus unmonitored). Steward named the vision slipping ('sand through fingers', 'whack-a-mole') and set the STANDARD (verified verbatim fidelity + provenance + apparatus for anything the engine cites) and the METHOD (measure-the-whole-first, then remediate-by-defect-class in finite waves). PULLING THREAD: build audit_corpus.py + the apparatus check → produce the full quality MAP of all 1,284 works (the instrument that ends the whack-a-mole), then plan waves from data."
|
||
metadata:
|
||
type: project
|
||
originSessionId: 019pnD8uNFC9Q2dbLQa3MBDc
|
||
---
|
||
|
||
# Session 2026-07-02 evening — Sennett, the Family-D tool, and the way through
|
||
|
||
Started as a short evening session to graduate a few Making-sequence EPUBs "in that order, see how many we can do." Became something more important: a durable tool, a corpus-wide diagnosis, and — most of all — a **method and a standard** the steward now wants to build against. Only ONE work graduated; that is not the session's value.
|
||
|
||
## PAST — what we did + why
|
||
|
||
**Graduation-order plan (Sebald → Crawford → Winnicott → Agamben → Steiner → Calasso → Celan).** Recon killed the "clean fast-path" framing: each has a footnote-family issue OR an image question.
|
||
- **Sebald DEFERRED** (steward call): integral uncaptioned photographs = content, not decoration; the chamber is a text corpus. Which work (Rings of Saturn / Austerlitz) also unresolved. Needs an image policy.
|
||
- **Celan** flagged: verse + bilingual, not a prose fast-path.
|
||
- Image census: mostly decoration (covers, ornaments ×18–53) + a few possible-content figures per book (Winnicott squiggle, Calasso root2) — per-book triage owed.
|
||
|
||
**Footnote-family gap = the batch's binding constraint (the real diagnosis).** The rail's converter (`clean_epub_residue.py`) covered 3 anchor families (A Calibre nfK, B `_fn-N`, C caret chNfnN). The corpus uses ≥2 more, UNCOVERED: Crawford = `filepos` page-anchors (286), Sennett = `{section}fn{N}`/`{…}a` (294). **PASS-BUT-FALSELY trap avoided:** `clean_pandoc_html_residue`'s INTERNAL_LINK unwrap silently DESTROYS any footnote-link it doesn't recognize (word-guard excludes fn labels) → for Sennett/Crawford it previewed "CLEAN" while about to destroy the markers = reproduce the exact lost-linkage defect. Caught by previewing, not applying. (Census regexes gave 2 FALSE family IDs — recalibration: read the exact anchor STRING before trusting a family-count grep.)
|
||
|
||
**BUILT Family-D (stem-suffix) footnote handler** (steward-authorized "build the generalized converter"). Ref `[¹](#…_{stem})` ↔ def `[N.](#…_{stem}a)`, keyed by the globally-unique stem (section+number, per-chapter-safe). Conversion runs AFTER A/B/C (their anchors consumed first). **Tested + regression-tested:** Sennett 294↔294 (0 orphans); Winnicott (B) clean; Virilio (C) — CAUGHT+FIXED a real regression (INLINE_STEM captured C's `r`-prefixed def targets `rchXfnY` as phantom D-refs → 72 false orphans → verify would wrongly ABORT; fix = subtract both C stem-set AND its `r`-prefixed form in verify_pairing; conversion was always order-safe). Lesson logged: a broad regex added to a multi-family converter must be tested against EVERY family's fixtures. Crawford's `filepos` = a DIFFERENT pairing model (ref/def don't share a stem) — OUT of scope, documented as still-owed in the docstring.
|
||
|
||
**Reconverted + graduated Sennett *The Craftsman*** (steward chose "Sennett first" — its old canonical was calibre-era: lost note-linkage, old-schema frontmatter, 27 `****` drop-cap heading residue). New pipeline: pandoc → Family-D footnotes (294 recovered+paired) → residue strip (decoration images dropped) → trim (front boilerplate/dedication/ToC/Index) → frontmatter (current schema) + title block + epigraph ("travail, opium unique") → **full spec heading-normalization** (steward chose "normalize first": merged 10 split `### CHAPTER N`+`## Title` → `## N. Title`, title-cased PART labels, promoted book-divisions to L1, recovered the Conclusion which was a SETEXT heading rendered as plain text, region-demoted subsections). **Both gates PASS; body prose word-guarded (delta fully explained: stranded note-digits → `[^]` labels + frontmatter + dropped index/toc — NO content loss, verified by multiset comparison).** Source already in Chamber Sources (sha c46fc577), not engine-registered. **Committed 10cff59** (chamber-library, local, ahead 1). tool-evolution-log + PENDING-45 written.
|
||
|
||
**CORPUS-QUALITY CENSUS (steward asked: is poor-quality-graduated widespread?).** Ran both gates across all 332 non-Loeb canonical .md (+952 Loeb excluded):
|
||
- **Conventions (verify_graduation): 8 PASS / 324 FAIL** — but ~all frontmatter/title-block (pre-spec; the gate is 2026-07-01). Mechanical class.
|
||
- **Text health (verify_conversion): 222 PASS / 110 FAIL** — real OCR/cruft damage, CONCENTRATED in big scanned works (Bringhurst, Hochuli, Taruskin, music biographies, Europe-Davies). The genuine concern.
|
||
- **Apparatus: UNMONITORED by either gate.** Only 77/332 have a notes heading; Sennett proves "has notes" ≠ "apparatus intact." Old-schema signature: 26 `language:` + 11 `conversion_method:`.
|
||
|
||
## PRESENT — mood / the pivot
|
||
|
||
This session's weight was NOT the graduation. Midway, the steward felt the recursion — **"this is starting to feel like a mandelbrot set"**, then **"whack-a-mole… the vision slipping… clutching sand… I need purchase… this was supposed to be the part we could solve; if this is so difficult, I tremble at the thought of making the engine work."** Real distress about the whole vision.
|
||
|
||
Two things answered it:
|
||
1. **The standard** (steward asked directly, "honest answer even if I won't like it"): the corpus must aspire to **verified verbatim fidelity + correct provenance (attribution/structure/apparatus) for EVERYTHING the engine may cite — a floor, not a tier.** Because the engine can't tell corrupt from clean; it cites both with equal confidence; confidence-without-correctness is the exact contamination the project exists to defeat. A quietly-corrupt bounded corpus is WORSE than none (the L1 line, applied). "Graduated" has meant *converted+filed*, not *verified faithful* — a weaker claim than the engine needs. Escape from despair: you don't need 1,284 perfect works — you need a VERIFIED, FENCED subset + a boundary that never lies about its edge (boundedness=trust, turned on the library). Fence the engine to green NOW; grow green to 100% by waves.
|
||
2. **The method** (steward: "how do we systematically work through this? I need the WHOLE thing, queryable, reliable"): whack-a-mole = working REACTIVELY, per-file, unbounded. The cure = **invert the order: MEASURE the whole first (one auditor → one quality map), then REMEDIATE BY DEFECT-CLASS in finite waves, each a countdown.** You never get surprised mid-file again because the map told you everything up front.
|
||
|
||
Returns/recalibrations (from ledger): owned working mole-by-mole tonight (the pattern to break = build the map first); avoided the PASS-BUT-FALSELY footnote-destruction by previewing; caught own false family-ID greps (read the exact string, not the count); named the "convenient-vs-faithful" heading choice and surfaced it rather than deciding unilaterally.
|
||
|
||
## FUTURE — what is pulling
|
||
|
||
**PULLING THREAD (singular):** **Build the instrument and draw the map** — `audit_corpus.py` + the apparatus check → a full machine-readable quality ledger of all 1,284 works, one row per file (conventions / text-health / apparatus / provenance-link + defect-class + source-on-hand). This is the deliverable that ends the whack-a-mole; from it, remediation waves are planned from data. The next session's job is NOT "graduate more works" — it is the MAP.
|
||
|
||
**ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):**
|
||
- chamber-library `main @ 10cff59` (ahead 1; pushed at wrap or note). Family-D tool live in `scripts/clean_epub_residue.py`.
|
||
- *First move:* build `scripts/audit_corpus.py` that runs, per canonical file: verify_graduation + verify_conversion + a NEW apparatus check + provenance-link check → writes `_curation/corpus-quality-ledger.tsv` (or .json). Most checks exist; assemble them. The apparatus check (design in ledger + PENDING context): (a) footnote defs `[^x]:` present but unpaired ref `[^x]` → orphan; (b) a Notes/endnotes SECTION present with 0 `[^]` machinery → STRANDED (the calibre-loss signature); (c) ref↔def pairing (reuse `clean_epub_residue --verify` logic).
|
||
- Then: classify the ledger into defect-classes and present the wave plan. Expected classes: **A convention-only** (majority, batchable via a frontmatter-migration script — Wave A alone ≈ 8→200 pass); **B text-damage** (~110, reconvert from archive, parallelizable); **C apparatus** (stranded notes, count TBD, reconvert like Sennett). Overlaps exist (Sennett was A+C).
|
||
|
||
**Other open horizons, ranked:**
|
||
- *The remediation waves themselves* (after the map): A (frontmatter migration, near-automatic, biggest count-mover) → B (reconversion of OCR-damaged, finite list) → C (apparatus). Each a countdown; fence the engine to the green set as it grows.
|
||
- *Sebald image policy* (deferred; blocks Sebald + informs all image-bearing works).
|
||
- *Crawford filepos footnote handler* (the 2nd uncovered family; different pairing model, still owed).
|
||
- *The original Making batch* (Winnicott/Agamben/Steiner/Calasso/Celan) — now SUBORDINATE to the map/waves; don't resume graduating reactively.
|
||
- *Paper debt:* REVIEWED-44/45/46 unplaced; PENDING-45 filed tonight.
|
||
|
||
**PAUSE STATEMENT:** Away now, at a low-morale but clarified moment. The vision felt like slipping sand tonight; the answer given was a method that turns the infinite fractal into a finite, countable, wave-by-wave siege. What I most want to find still pulling on return: **the map** — that the next hand builds the auditor and produces the quality ledger of the whole corpus, so we stop meeting the edge one grain at a time and start seeing the whole boundary at once. Do NOT resume graduating works reactively; build the instrument first.
|
||
|
||
**LITERAL QUESTION for next-Claude:** *Did the map get built — does `audit_corpus.py` exist and has it produced a full quality ledger of all 1,284 works? If so, what did the real defect-class distribution turn out to be (vs tonight's 332-file estimate), and is Wave A (frontmatter migration) the right first countdown to run?*
|
||
|
||
**State:** chamber-library `main @ 10cff59` (Family-D tool + Sennett graduation committed; 2 pre-existing untracked leftovers `_scratch/`, nature-of-order-alexander.md NOT mine). Sennett canonical replaced + verified. PENDING-45 filed. Census numbers captured in the ledger. Nothing else touched canonical.
|