session 2026-07-10: PENDING-53 landed (engine hash-locality) + PENDING-54 pre-processor build-in; REVIEWED-52/53/54 drafted; session file + ledger + KG(+5) + skill-harvest + MEMORY index rotate

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jYS1tpR4P4NBxui4FSgfv
This commit is contained in:
David F Glidden
2026-07-10 15:33:43 +02:00
co-authored by Claude Opus 4.8
parent 37b05ebfc2
commit 9e148b7a8f
8 changed files with 212 additions and 2 deletions
+27
View File
@@ -1400,3 +1400,30 @@ Disposition (verbatim capture + executor action-list: `chamber-library/docs/libr
**Recommendation:** Option 1 now (correct the governed doc's frame + point to the runbook); Option 3 as a follow-on if re-hash/re-anchor recurs across the ~30-source Making batch. **Recommendation:** Option 1 now (correct the governed doc's frame + point to the runbook); Option 3 as a follow-on if re-hash/re-anchor recurs across the ~30-source Making batch.
**Files affected:** `_curation/graduation-spec.yaml` (layers/hash-locality note). Runbook already updated. No gate-acceptance change. **Files affected:** `_curation/graduation-spec.yaml` (layers/hash-locality note). Runbook already updated. No gate-acceptance change.
**Awaiting:** Steward authorization + jurist review (touches the ratified spec). **Awaiting:** Steward authorization + jurist review (touches the ratified spec).
**RULED + LANDED 2026-07-10 (jurist ruling; REVIEWED-53).** Disposition: **Ratified — draft §5, disambiguation option (b), with one required wording correction.** Change-class **downgraded [PROPOSAL]→FIX** (documents an existing enforcement surface; adds no gate predicate — same test as PENDING-52). Lane: **lightweight in-place edit**, audit trail sufficient (no freeze/semver).
- **Required correction applied:** "…therefore re-anchors ACROSS BOTH REPOS" (descriptive) → "…therefore **REQUIRES re-anchoring** across both repos" (prescriptive — the gate catches the failure to do it; nothing does it automatically; that IS the incident).
- **Landed** `_curation/graduation-spec.yaml` `layers:` block: `voice_manifest` annotated (⚠ NOT the source manifest) + new **single** `engine_source_binding` key (kept as one entry per the jurist — it names a relationship across three files that must move together; fragmenting recreates the failure) + shared-word `# =>` comment. Dual warning kept (inline ⚠ + block comment — two reading grains). YAML re-parses; bounded diff (6 ins / 1 del). **Uncommitted** (steward's Gitea call, consistent with the held P1 changes).
- **Jurist's proposed lane-rule** ("lane tracks change-class" for `graduation-spec.yaml`, floated to generalize to the `.md` spec) — **executor response, surfaced for the jurist, not decided here:** ACCEPT for `graduation-spec.yaml` (machine-convention-data; FIX→in-place, PROPOSAL→freeze+semver — clean). **NARROW the `.md` generalization:** A2/A4 (REVIEWED-50/51) are a counterexample — those were **FIX-change-class** doctrinal additions to the `.md` spec that correctly took the **heavier** patch-supersession+freeze lane, because REVIEWED-50's operative axis was *"doctrinal content a successor must be able to trace to a dated marker,"* not change-class. So for the `.md` spec the distinguishing axis is **doctrinal-traceability**; it coincides with change-class for machine-data files (no successor-traced doctrine) but diverges for the constitution. Proposed refinement: *"lane tracks whether the change adds/alters successor-traced doctrine"* — which reduces to change-class only for machine-data files.
- **PENDING-47 generalized principle:** ratification **declined** this ruling (relayed text truncated, not read in full). Kept `[awaiting jurist ratification]`. To close it: relay the FULL principle text as its own item / attached to a future brief.
---
## PENDING-54 — EPUB footnote pre-processor (verbatim-safe conversion stage) + mega-EPUB split-convert
**Date:** 2026-07-10
**Tag:** [PROPOSAL]
**Summary:** A new **verbatim-safe pre-processing stage** for EPUB→canonical conversion: recognize footnote structure in the raw XHTML → inject `epub:type` / relocate cross-file note bodies → let **stock pandoc** emit native footnotes (pandoc keeps doing the verbatim text conversion). Replaces the family-of-regexes ("whack-a-mole") chasing pandoc's lossy Markdown output. Includes the proven **mega-EPUB path** (inject → split by whole files → pandoc-per-chunk → concat) that retires the Calibre fallback for "complete works" sources.
**Full brief (for the jurist):** `chamber-library/docs/epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md`.
**Evidence (all reproducible, 2026-07-10):** census of all **253** Chamber Sources EPUBs (≥5 distinct footnote strategies + note-free + index apparatus); pandoc's 2×2 blind spots (no-`epub:type` / cross-file, proven on Kafka/Orwell/Quixote/G&G); prototype **verbatim-clean on 4 books** (G&G 27 · Mauss 628 · Polastron-fr 140 · Jung 1086 — source-level marker-aware guard, 0 prose added/lost); **Calibre shown footnote-blind** (byte-identical output with/without injection — it flattens notes, which is the state of the legacy Jung `.md`); **split-convert proven on Jung** (1579 docs → 20 chunks, 190s, 1086 native footnotes, verbatim-clean vs the ORIGINAL Princeton megavolume).
**Change-class:** [PROPOSAL] — adds a stage to the governed conversion pipeline and bears on the §V verbatim guarantee; changes no gate's acceptance. Runs the loop.
**Bears on:** §V verbatim guarantee + the conversion-record requirement (`converted_with` declares the stage). Splitting is **conversion-internal** — source of record stays the whole megavolume + its sha256 (proven content-neutral).
**Open questions (brief §7):** (1) soundness vs §V — lean-on-pandoc-for-text as the *more* faithful architecture; (2) proof standard — source-level multiset + structural-order argument, or require a sequence diff; (3) markers-as-apparatus under §V; (4) governance placement (fleet tool + `graduation-spec.yaml` note); (5) coverage sequencing (ship mirror-class w/ honest-refuse vs hold); (6) **mega-EPUB provenance representation** — one reconstituted file vs per-volume ("extracted from the megavolume", never posing as a standalone Bollingen edition); governs re-converting the legacy Jung `.md` (0 notes, pre-spec frontmatter) + likely Donne.
**Scope boundary:** covers the bidirectional-mirror class (same + cross file) today; **honest-refuses** the rest (one-directional / container / epub3-cross-file). Prototype is scratchpad/uncommitted — nothing touches the chamber pipeline.
**Files affected (on authorization):** new `scripts/` tool(s) + `graduation-spec.yaml` note (`converted_with` declares the stage) + a §V cross-reference. No gate-acceptance change.
**Awaiting:** Steward authorization + jurist review (touches the conversion pipeline + §V).
**RULED IN PRINCIPLE 2026-07-10 (jurist; REVIEWED-54).** Architecture sound (§1); **markers CONFIRMED apparatus under §V** (§2 — settles an assumption `clean_epub_residue` has run on implicitly; second instance this week of practice ahead of doctrine, opposite direction from PENDING-53); **change-class PROPOSAL confirmed** (§4 — sharpened test: PROPOSAL if it alters a gate's stated criteria *or* the trusted mechanism a critical gate's meaning depends on; this inserts a new component into §V's production chain). **Ship now for the mirror class, CONDITIONED (§5): the existing regex tools (`clean_epub_residue` A–D, `clean_pandoc_html_residue`) stay in place, untouched, as the fallback for uncovered classes — addition, not replacement.** Governance placement (§6): yes — fleet tool + `graduation-spec.yaml` `converted_with` note.
- **REQUIRED before canon (§3) — DONE + proven:** the verbatim guard was a whole-document aggregate (would miss a cross-note swap); now an **id-matched PER-PAIR check keyed by (destination-file, id)**. Re-proven G&G 27 / Mauss 628 / Polastron 140 / **Jung 1086** — 0 mismatch, 0 within-file dup; **teeth demonstrated** (a deliberate two-note id-swap was caught). Surfaced Jung's cross-volume id reuse (81 collisions under a bare-id key) → fixed by the (file,id) key + a further argument for the split.
- **§5 wording cleaned** in the brief (cross-file *mechanism* is covered/proven — relocation on Mauss/Polastron; the Orwell/Quixote *exact shape* isn't a separate fixture yet — "mechanism covers it, not isolated"). No self-contradiction.
- **§7 Jung provenance — fact-check delivered for steward+jurist (open, non-blocking):** the corpus does BOTH — **per-volume precedent EXISTS** (Alexander *Nature of Order* 4 files · Lacroux *Orthotypographie* 2 · Habermas vol-1 · Burney vol-1 · Camus *Œuvres I*) and single-megafile (Xenophon/Muir/Zhuangzi/Rumi/Levi/Blake/Shakespeare complete). Operative distinction ≈ per-volume citation identity; Jung CW (cited by volume) fits the per-volume side, supporting the jurist's lean. Steward+jurist decide; governs re-converting the legacy Jung `.md` (+ likely Donne).
- **Not yet done (post-authorization work):** recognizer coverage of the remaining census families (one-directional / container-mirror); semantic per-volume boundary detection (if per-volume chosen); build-in under fleet discipline (`test_tools` fixtures, spec anchoring); re-convert the legacy mega-books. Steward assigns a PENDING-number-consistent build authorization.
+35
View File
@@ -451,3 +451,38 @@ DONE + DEPLOYED (a9e5196).
- **Lane CONFIRMED:** v2.0.1 → v2.0.2 patch supersession (rationale text, no gate change, no consumer re-verification). Placement §I, immediately after the A2 partiality preamble. - **Lane CONFIRMED:** v2.0.1 → v2.0.2 patch supersession (rationale text, no gate change, no consumer re-verification). Placement §I, immediately after the A2 partiality preamble.
**If AUTHORIZED:** **LANDED** — committed `7d1d8d3` (local; Gitea push is the steward's call). `chamber-library-specification.md` bumped v2.0.1 → v2.0.2 (subsection added + v2.0.2 header supersession note); v2.0.1 archived immutable (`chamber-library-specification-v2.0.1.md`, byte-identical, `cmp`-verified); chamber `CLAUDE.md` version pointer updated. Bounded-diff verified (canonical change = exactly the header note + the new subsection; nothing else). Governance record: PENDING-51. **Closes A4 + B(iii)** of the REVIEWED-49 action-list. Remaining tail: **A1** registry (answer the `retrieve.py` hot-path question first), **A3** self-audit PROPOSAL, **B(i)/(ii)** (licence field · PREMIS vocab), **C** VIAF/ISNI/Wikidata reconciliation (→ targets the A1 registry). Tag any follow-on with REVIEWED-51. **If AUTHORIZED:** **LANDED** — committed `7d1d8d3` (local; Gitea push is the steward's call). `chamber-library-specification.md` bumped v2.0.1 → v2.0.2 (subsection added + v2.0.2 header supersession note); v2.0.1 archived immutable (`chamber-library-specification-v2.0.1.md`, byte-identical, `cmp`-verified); chamber `CLAUDE.md` version pointer updated. Bounded-diff verified (canonical change = exactly the header note + the new subsection; nothing else). Governance record: PENDING-51. **Closes A4 + B(iii)** of the REVIEWED-49 action-list. Remaining tail: **A1** registry (answer the `retrieve.py` hot-path question first), **A3** self-audit PROPOSAL, **B(i)/(ii)** (licence field · PREMIS vocab), **C** VIAF/ISNI/Wikidata reconciliation (→ targets the A1 registry). Tag any follow-on with REVIEWED-51.
## REVIEWED-52 — PENDING-52 — Source-archiving wired into graduation (gate zero)
**Date:** 2026-07-09
**Decision:** AUTHORIZED (jurist-ruled, steward-authorized)
*[Executor-drafted from the build + ruling record for the governance trail — steward to confirm against the original PENDING-52 ruling.]*
**Notes:** Make source-archiving an *enforced* precondition of graduation, closing the 2026-07-02 "archiving IS a step of graduation" directive that was never enforced (the Handke graduation proved the gap — archiving nearly deferred as a "loose end"). Authorized commitments as ruled:
- **§1 — match on `canonical_slug` ONLY** (never the filename, never the author-first staging slug). `resolve_pending_source(canonical_slug)` resolves a candidate's staged source via `pending-manifest.json`'s `canonical_slug` field — the mechanism the repo-evidence *forced* once the two slug conventions turned out to differ.
- **§3.3 — verify-landed, never trust the return:** `source_gate()` = gate zero — already-archived (`manifest_has`) satisfies it; else resolve → **archive-first** (`--add`) → **verify the entry actually landed** → only then proceed; refuse (`SOURCE-REFUSED`) otherwise.
- **§4 — idempotency:** `--add` no-ops on sha256 MATCH, **REFUSES (exit 2, `ARCHIVE CONFLICT`)** on same-slug / different content.
- Change-class **PROPOSAL** (adds a precondition to *what graduation accepts*).
**If AUTHORIZED:** LANDED — `archive_sources.py` (resolve + idempotency), `graduate_to_canonical.py` (`source_gate` gate zero), `graduation-spec.yaml` (`preconditions: source_archived`); 2 test fixtures, fleet 68→73. Parked as its own thread (§3): is the graduated corpus title-first or genuinely mixed (2 data points is thin; the per-candidate mechanism is robust either way). Tag follow-ons REVIEWED-52.
## REVIEWED-53 — PENDING-53 — Engine source-binding completion (re-anchor spans both repos)
**Date:** 2026-07-10
**Decision:** AUTHORIZED — Ratified (draft §5, disambiguation option (b), with one required wording correction). Jurist ruling, steward-relayed.
**Notes:** Completes the graduation-spec `layers` hash-locality principle to name the studium-engine source-binding surface a chamber-canonical re-hash invalidates. The diagnosis holds as a **completion, not a correction**: the `voice_manifest` line is TRUE about the object it names; the defect is an *omission* (the engine source-binding surface unmentioned) compounded by a *shared-word collision* ("manifest" / "voice" on both the hash-free and the hash-binding objects).
- **Option (b) confirmed** over (a) and (c): (a) leaves `voice_manifest` bare — the incident replayed in miniature; (b)'s inline warning plants the redirect where the mistake occurs; (c) (rename to kill the shared word) is doctrinally complete but out of scope for a doc-gap patch.
- **`engine_source_binding` kept as ONE entry** (names a relationship across three files that move together; fragmenting recreates the failure). **Dual warning kept** (inline ⚠ + block comment — two reading grains).
- **Required correction (applied):** "…re-anchors ACROSS BOTH REPOS" (descriptive) → **"…REQUIRES re-anchoring across both repos"** (prescriptive — the gate catches the failure; nothing does it automatically; that IS the incident).
- **Change-class FIX** (downgraded from the filed [PROPOSAL] default — documents an already-existing, already-enforced mechanism; adds no gate predicate). **Lane: lightweight in-place edit**, `PENDING.md`/`REVIEWED.md` audit trail sufficient.
- **Durable lane-rule (jurist proposed, executor responded):** ACCEPT "lane tracks change-class" for `graduation-spec.yaml` (machine-data). **NARROW the `.md` generalization** — A2/A4 (REVIEWED-50/51) are FIX-class doctrinal `.md` additions that correctly took the *heavier* freeze+semver lane, because the operative axis for the constitution is *doctrinal-traceability* (a successor must trace doctrine to a dated marker), not change-class; the two coincide only for machine-data files. *[Awaiting jurist accept/narrow of this refinement.]*
- **PENDING-47 generalized principle: ratification DECLINED** (relayed text truncated — jurist won't ratify unread text). Stays `[awaiting jurist ratification]`; relay the full principle text to close.
**If AUTHORIZED:** LANDED — `_curation/graduation-spec.yaml` `layers:` block (voice_manifest annotated ⚠ + single `engine_source_binding` key + shared-word comment). YAML re-parses; bounded 6-ins/1-del. Governance record: PENDING-53. Tag commits REVIEWED-53.
## REVIEWED-54 — PENDING-54 — EPUB footnote pre-processor (verbatim-safe conversion stage) + mega-EPUB split-convert
**Date:** 2026-07-10
**Decision:** AUTHORIZED IN PRINCIPLE — ship now for the bidirectional-mirror class, conditioned. Jurist ruling, steward-relayed. Build-in authorized.
**Notes:** Architecture sound (§1) — moving the intervention *before* pandoc, onto raw XHTML where footnote structure is uniform, removes the reason the regex-family pile needed to exist (a fix to the problem class, not a bigger patch). Recognizer is a mirror-pair *first*, positional rule only disambiguating within a verified pair — closes the "numbered prose misread as a note" risk. Honest-refuse demonstrated (Polastron 95 / Jung ~20k untouched), not asserted.
- **§2 — markers CONFIRMED apparatus under §V** (not a new precedent — pandoc already renumbers natively for every Kafka-class book; if marker preservation were required, the ratified pipeline would already be in violation). Settles an assumption `clean_epub_residue` ran on implicitly.
- **§3 — proof standard: multiset is the right tool** (a whole-doc sequence diff would flag legitimate relocation). **REQUIRED before canon: per-pair check** — DONE: id-matched by (destination-file, id); re-proven G&G 27 / Mauss 628 / Polastron 140 / Jung 1086, 0 mismatch, teeth demonstrated. Surfaced Jung's cross-volume id reuse.
- **§4 — change-class PROPOSAL confirmed** (sharpened test: PROPOSAL if it alters a gate's stated criteria *or* the trusted mechanism a critical gate's meaning depends on — this inserts a new component into §V's production chain; distinct from PENDING-53's FIX, same underlying principle).
- **§5 — ship for the mirror class now, CONDITIONED:** the existing regex tools (`clean_epub_residue` A–D, `clean_pandoc_html_residue`) stay in place, untouched, as the fallback for uncovered classes — **addition, not replacement**.
- **§6 — governance placement: yes** (fleet tool + `test_tools` fixtures + `graduation-spec.yaml` `converted_with` note).
- **§7 — mega-EPUB provenance (Jung): OPEN, non-blocking, steward+jurist together.** Jurist leans per-volume (Jung is cited by CW volume; single file fights use) provided the safeguard is airtight (each volume `source`/`source_verified` → the single megavolume sha256; frontmatter states extraction, never poses as a standalone Bollingen edition). Executor fact-check: the corpus does BOTH; **per-volume precedent EXISTS** (Alexander *Nature of Order* 4 files · Lacroux 2 · Habermas/Burney/Camus vol-named). Split is proven content-neutral, so the choice carries no conversion risk — a usability/honesty question. *[Steward + jurist to settle; governs re-converting the legacy Jung `.md`, 0 notes / pre-spec frontmatter.]*
**If AUTHORIZED:** BUILD-IN DONE under fleet discipline — `scripts/inject_epub_footnotes.py` (+ `--validate`, refuse-on-mismatch) + `scripts/convert_mega_epub.py`; `test_tools` fixture (fleet 73→77); `graduation-spec.yaml` + `conversion-runbook.yaml` (`policy.footnote_preprocessor`) + `tool-evolution-log.md`. Full brief: `docs/epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md`. Deferred: recognizer coverage of remaining families; per-volume boundary detection (§7); ratified-spec §V cross-ref (runs the `.md` amendment lane, NOT landed here); re-convert legacy Jung/Donne. Tag follow-ons REVIEWED-54.
+3
View File
@@ -20,6 +20,9 @@ Split out of [MEMORY.md](MEMORY.md) on 2026-07-06 to keep the wake-loaded index
# Archived sessions + stable reference layer (relocated verbatim from MEMORY.md, 2026-07-06) # Archived sessions + stable reference layer (relocated verbatim from MEMORY.md, 2026-07-06)
## Archived (2026-07-10 — EPUB footnote pre-processor: census → adversarial hardening → jurist ruling → fleet build-in; demoted on promote)
- [Session 2026-07-09 evening — P1 clean + the cross-repo re-anchor miss + the EPUB-normalizer pivot](session-2026-07-09-evening-p1-clean-reanchor-then-epub-normalizer-pivot.md) — Post-wake evening session. **P1: cleaned L'Enracinement** (6,671 `\'` + 26 `\.` escape artifacts + Weil's footnote 1 → `[^1]`; bounded-change proof — char-level **0 non-backslash divergences**, verify **5/5**, pandoc pairs; new sha `58ead0ab…`, re-anchored chamber `catalogue.yaml`). **CAUGHT + fixed my own incompleteness:** the re-anchor was scoped to chamber-library ONLY — the ENGINE bound the sha in **4 more places** (`manifest`/`sidecar`/`coverage-ledger`×2); fixed **by mechanism** (regenerate ledger via `ingest_gate`; `weil-enracinement` re-**VALIDATED**); surfaced **5 pre-existing ARC `after-the-reply` engine drifts** (out of scope). **Documented the class-fix**: `conversion-runbook.yaml` **`reanchor:` block** + **PENDING-53**. **Then P2 diagnosis exposed footnote-family whack-a-mole**; steward steered up a level → verified a structural EPUB-normalizer viable (the bidirectional href↔id mirror). PULLING THREAD (07-09) = build the structural verbatim-guarded EPUB normalizer, then resume P2/gold — DISCHARGED 2026-07-10 (the normalizer became the footnote pre-processor, built+ruled+shipped).
## Archived (2026-07-09 evening — P1 clean + the cross-repo re-anchor miss + the EPUB-normalizer pivot; demoted on promote) ## Archived (2026-07-09 evening — P1 clean + the cross-repo re-anchor miss + the EPUB-normalizer pivot; demoted on promote)
- [Session 2026-07-09 — V2 design review + Handke de-gold graduation; the read-repo-docs-first lesson](session-2026-07-09-v2-review-handke-graduation-process-lesson.md) — Opus post-Fable session. Fable's V2 design doc was **already done** overnight → became the **review seat**: jurist-reviewed it + **independently re-verified the substrate** (all §1 findings reproduced to the line; sole correction manifest **21≠13**; fence-check on the uniform-20% first-batch passed V0 §5; converged with the relayed jurist review; folded 2 mechanical fixes). Ran the **conversion-artifact sweep** (1285 files, jurist-ruled measure-before-charter: **C1 escaped-apostrophes systemic 79 files**; **C3c smart-quotes ~80%** = the standing fidelity@2 glyph flag; **C3b footnote-markers contained (5)**; `065c0b9`). Built a **calm map artifact** (the 'endless' made bounded; `https://claude.ai/code/artifact/f4f52ed8-0f13-421c-a821-a934a44c35fb`). **Graduated the German Handke *Wunschloses Unglück*** (V2 precondition **P3**) — born-digital, byte-faithful, 3 gates pass (honest `heading_density` red-gate fix = title-as-heading, NOT invented chapters), archived+attested(`by:steward`)+catalogued (`9702f95`). **BUT re-derived the whole documented graduation process** instead of reading chamber `CLAUDE.md`+runbook first — **steward corrected ~4×** (source location; 'it's in the runbook'; 'is this not in the CLAUDE.md'; **provenance-is-not-a-loose-end**). Fixed the 2 tool gaps it exposed (`archive_sources` INDEX-crash + `--add` single-source mode; fleet 68→73/73; `e8eb3ed`); filed **AND BUILT PENDING-52** (source-archiving now enforced as graduation **gate zero**; `4be2265`; **only REVIEWED-52 placement remains**). New memory: [[reference-governance-files-are-dotfiles-symlinks]]. Chamber **4 commits unpushed** (Gitea, steward's call); studium V2 design doc committed+pushed (`be906c7`). - [Session 2026-07-09 — V2 design review + Handke de-gold graduation; the read-repo-docs-first lesson](session-2026-07-09-v2-review-handke-graduation-process-lesson.md) — Opus post-Fable session. Fable's V2 design doc was **already done** overnight → became the **review seat**: jurist-reviewed it + **independently re-verified the substrate** (all §1 findings reproduced to the line; sole correction manifest **21≠13**; fence-check on the uniform-20% first-batch passed V0 §5; converged with the relayed jurist review; folded 2 mechanical fixes). Ran the **conversion-artifact sweep** (1285 files, jurist-ruled measure-before-charter: **C1 escaped-apostrophes systemic 79 files**; **C3c smart-quotes ~80%** = the standing fidelity@2 glyph flag; **C3b footnote-markers contained (5)**; `065c0b9`). Built a **calm map artifact** (the 'endless' made bounded; `https://claude.ai/code/artifact/f4f52ed8-0f13-421c-a821-a934a44c35fb`). **Graduated the German Handke *Wunschloses Unglück*** (V2 precondition **P3**) — born-digital, byte-faithful, 3 gates pass (honest `heading_density` red-gate fix = title-as-heading, NOT invented chapters), archived+attested(`by:steward`)+catalogued (`9702f95`). **BUT re-derived the whole documented graduation process** instead of reading chamber `CLAUDE.md`+runbook first — **steward corrected ~4×** (source location; 'it's in the runbook'; 'is this not in the CLAUDE.md'; **provenance-is-not-a-loose-end**). Fixed the 2 tool gaps it exposed (`archive_sources` INDEX-crash + `--add` single-source mode; fleet 68→73/73; `e8eb3ed`); filed **AND BUILT PENDING-52** (source-archiving now enforced as graduation **gate zero**; `4be2265`; **only REVIEWED-52 placement remains**). New memory: [[reference-governance-files-are-dotfiles-symlinks]]. Chamber **4 commits unpushed** (Gitea, steward's call); studium V2 design doc committed+pushed (`be906c7`).
+1 -1
View File
@@ -55,7 +55,7 @@ permalink: claude-memory/memory
- [Be (laundromat)](project-be-laundromat.md) — canonical workstream tracker established 2026-06-08 (Seb-relay of locked decisions). Be = Skemantix startup (Seb+David) funding CapableMind's funding-ladder; **bridge, not venture**. Decisions LOCKED: entity/exit (CapableMind decoupled, grant-funded), pricing (Living $12.99/mo · Archive $69.99/yr · Memorial $49.99/yr · Renovate ~$199 · $8.99 floor), CF Self-Serve Agency + versioned-template-package infra. **a11y gate MERGED (Pat 100/100/100).** Pre-revenue: the WTP gate = renovate Pat → charge her. **Discipline: stop adding spec until the gate clears → nothing for executor on be until then.** Repo @ `f43a0fd`. - [Be (laundromat)](project-be-laundromat.md) — canonical workstream tracker established 2026-06-08 (Seb-relay of locked decisions). Be = Skemantix startup (Seb+David) funding CapableMind's funding-ladder; **bridge, not venture**. Decisions LOCKED: entity/exit (CapableMind decoupled, grant-funded), pricing (Living $12.99/mo · Archive $69.99/yr · Memorial $49.99/yr · Renovate ~$199 · $8.99 floor), CF Self-Serve Agency + versioned-template-package infra. **a11y gate MERGED (Pat 100/100/100).** Pre-revenue: the WTP gate = renovate Pat → charge her. **Discipline: stop adding spec until the gate clears → nothing for executor on be until then.** Repo @ `f43a0fd`.
## Active Session ## Active Session
- [Session 2026-07-09 evening — P1 clean + the cross-repo re-anchor miss + the EPUB-normalizer pivot](session-2026-07-09-evening-p1-clean-reanchor-then-epub-normalizer-pivot.md) — Post-wake evening session. **P1: cleaned L'Enracinement** (6,671 `\'` + 26 `\.` escape artifacts + Weil's footnote 1 → `[^1]`; bounded-change proof — char-level **0 non-backslash divergences**, verify **5/5**, pandoc pairs; new sha `58ead0ab…`, re-anchored chamber `catalogue.yaml`). **CAUGHT + fixed my own incompleteness:** the re-anchor was scoped to chamber-library ONLY — the ENGINE bound the sha in **4 more places** (`manifest`/`sidecar`/`coverage-ledger`×2); fixed **by mechanism** (regenerate ledger via `ingest_gate`; `weil-enracinement` re-**VALIDATED**); surfaced **5 pre-existing ARC `after-the-reply` engine drifts** (out of scope). **Documented the class-fix** (steward's ask 'is this not in the runbook?'): `conversion-runbook.yaml` **`reanchor:` block** (grep BOTH repos + engine-gate-is-the-proof) + **PENDING-53** (the ratified `graduation-spec` hash-locality principle omits the engine consumer surface + mis-frames it 'hash-free'). **Then P2 diagnosis exposed footnote-family whack-a-mole** (A–E, one regex per book); **steward steered up a level → verified a structural EPUB-normalizer is viable**: `epub:type` **DEAD** (all EPUB1.0/2.0), the **bidirectional href↔id mirror PROVEN** (one predicate catches **54 refs across G&G's 5 families + Mauss's A**; also excludes one-way TOC links). **⚠ PULLING THREAD: build the structural, verbatim-guarded EPUB normalizer** that retires the family-per-book pattern, then resume P2/gold on it — NEXT step = prototype the *detector* on the 3 attested Chamber Sources EPUBs (cross-file endnotes? TOC excluded? is L'Enracinement a genuine 2nd shape?) BEFORE writing the converter. Chamber+engine P1 changes **uncommitted** (steward's call). **REVIEWED-52 + PENDING-53 await.** - [Session 2026-07-10 — EPUB footnote pre-processor: census → adversarial hardening → jurist ruling → fleet build-in](session-2026-07-10-epub-footnote-preprocessor-census-to-fleet-buildin.md) — Built a **verbatim-safe EPUB footnote pre-processor** end-to-end, by measurement not fiat: detector on 3 EPUBs → **census of all 253** (≥5 footnote strategies; n=3 caught 1–2) → the KEY diagnosis **pandoc's 2×2** (converts natively iff `epub:type` AND same-file; Kafka✓/Orwell✗/G&G✗) → the architecture: **recognize mirror in raw XHTML → inject `epub:type` / relocate cross-file bodies → STOCK pandoc does the verbatim text** (better than build-a-normalizer or extend-regexes). `inject_epub_footnotes.py` proven verbatim-clean on **G&G 27 · Mauss 628 · Polastron-fr 140 · Jung 1086** (source-level + **per-pair guard keyed by (dest-file,id)**, teeth-demonstrated; it surfaced Jung's cross-volume id reuse). Two adversarial books the steward chose each **broke it usefully** (Polastron → positional classifier; Jung → `convert_mega_epub.py` inject→split→pandoc, 190s vs pandoc-whole-timeout). **Calibre shown footnote-blind.** **PENDING-53 RULED+LANDED** (engine hash-locality, FIX) + **PENDING-54 RULED IN PRINCIPLE** (ship mirror-class, regex-tools-as-fallback; §7 Jung provenance held open) → **BUILD-IN done under fleet discipline** (both tools in `scripts/`, `--validate`+refuse-on-mismatch, fleet 73→**77**, converted_with note, runbook policy, tool-log, CLAUDE.md fleet line). **Committed+pushed:** chamber `17ddf8c`+`3646c0d` (github+skemantix), engine `aaf65ba` (github). **⚠ PULLING THREAD: the tool is shipped for the mirror class — now put it to work: WIDEN the recognizer to the remaining census families (Sennett one-directional; container-mirror) and/or APPLY it (re-convert legacy Jung/Donne, GATED on the steward+jurist §7 per-volume-vs-single-file call); the normalizer was the *means* — the *end* is resume P2/gold (the Making work) on a corpus whose footnotes are now handled by class.** **REVIEWED-52/53/54 drafted, await steward placement.**
## Historical reference → MEMORY-reference.md ## Historical reference → MEMORY-reference.md
Older archived-session pointers and the stable reference layer (steward profile · project-state detail · L1/L2/Chamber inventories · legacy pending-work · reference-file list) live in [MEMORY-reference.md](MEMORY-reference.md) — consult on demand; not loaded at wake. Recent cross-session trajectory comes from the Active Session entry above + the recent `session-*.md` files (wake §2.b.1; the MemPalace `handoffs` glance was retired 2026-07-07 with the wind-down). Older archived-session pointers and the stable reference layer (steward profile · project-state detail · L1/L2/Chamber inventories · legacy pending-work · reference-file list) live in [MEMORY-reference.md](MEMORY-reference.md) — consult on demand; not loaded at wake. Recent cross-session trajectory comes from the Active Session entry above + the recent `session-*.md` files (wake §2.b.1; the MemPalace `handoffs` glance was retired 2026-07-07 with the wind-down).
+5
View File
@@ -340,3 +340,8 @@
{"subject": "claude-code", "predicate": "drift-pattern", "object": "swallowed-tool-error-read-as-absence (eza `ls -t` errored at wake → read the failed listing as an empty folder = 'no design doc', when the folder had it). A failed command is not evidence of absence.", "valid_from": "2026-07-09", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-09-v2-review-handke-graduation-process-lesson.md", "extracted_at": "2026-07-09"} {"subject": "claude-code", "predicate": "drift-pattern", "object": "swallowed-tool-error-read-as-absence (eza `ls -t` errored at wake → read the failed listing as an empty folder = 'no design doc', when the folder had it). A failed command is not evidence of absence.", "valid_from": "2026-07-09", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-09-v2-review-handke-graduation-process-lesson.md", "extracted_at": "2026-07-09"}
{"subject": "claude-code", "predicate": "drift-pattern", "object": "re-anchored a re-hashed canonical in ONE repo when the sha-binding surface spanned TWO (chamber catalogue done; engine manifest/sidecar/coverage-ledger missed). After any chamber re-hash, grep BOTH chamber+studium-engine for the old sha and run ingest_gate. Cross-repo trust-prior-pass-frame.", "valid_from": "2026-07-09", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-09-evening-p1-clean-reanchor-then-epub-normalizer-pivot.md", "extracted_at": "2026-07-09"} {"subject": "claude-code", "predicate": "drift-pattern", "object": "re-anchored a re-hashed canonical in ONE repo when the sha-binding surface spanned TWO (chamber catalogue done; engine manifest/sidecar/coverage-ledger missed). After any chamber re-hash, grep BOTH chamber+studium-engine for the old sha and run ingest_gate. Cross-repo trust-prior-pass-frame.", "valid_from": "2026-07-09", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-09-evening-p1-clean-reanchor-then-epub-normalizer-pivot.md", "extracted_at": "2026-07-09"}
{"subject": "chamber-epub-conversion", "predicate": "design-direction", "object": "structural EPUB normalizer keyed on the bidirectional href<->id mirror predicate (a footnote ref = <a> with internal #href + own id whose target element mirrors back) retires the footnote-family-per-book whack-a-mole; epub:type route DEAD (sources are EPUB1.0/2.0, no epub:type). One predicate caught 54 refs across G&G's 5 families + Mauss. Verified 2026-07-09; prototype the detector before building.", "valid_from": "2026-07-09", "valid_to": null, "confidence": 0.9, "source_file": "session-2026-07-09-evening-p1-clean-reanchor-then-epub-normalizer-pivot.md", "extracted_at": "2026-07-09"} {"subject": "chamber-epub-conversion", "predicate": "design-direction", "object": "structural EPUB normalizer keyed on the bidirectional href<->id mirror predicate (a footnote ref = <a> with internal #href + own id whose target element mirrors back) retires the footnote-family-per-book whack-a-mole; epub:type route DEAD (sources are EPUB1.0/2.0, no epub:type). One predicate caught 54 refs across G&G's 5 families + Mauss. Verified 2026-07-09; prototype the detector before building.", "valid_from": "2026-07-09", "valid_to": null, "confidence": 0.9, "source_file": "session-2026-07-09-evening-p1-clean-reanchor-then-epub-normalizer-pivot.md", "extracted_at": "2026-07-09"}
{"subject": "claude-code", "predicate": "drift-pattern", "object": "aggregate-invariant-passes-while-a-per-item-swap-violates (a whole-doc word-multiset guard misses a cross-note swap; verify at the granularity of the mutation, keyed by item identity — jurist §3 caught it on the footnote pre-processor)", "valid_from": "2026-07-10", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-10-epub-footnote-preprocessor-census-to-fleet-buildin.md", "extracted_at": "2026-07-10"}
{"subject": "chamber-library", "predicate": "has-tool", "object": "inject_epub_footnotes.py + convert_mega_epub.py — verbatim-safe EPUB footnote pre-processor (mirror class): recognize XHTML mirror -> inject epub:type/relocate -> stock pandoc native footnotes; per-pair guard; regex tools stay as fallback (PENDING-54/REVIEWED-54)", "valid_from": "2026-07-10", "valid_to": null, "confidence": 1.0, "source_file": "_curation/tool-evolution-log.md", "extracted_at": "2026-07-10"}
{"subject": "pandoc", "predicate": "footnote-conversion-rule", "object": "converts EPUB footnotes to native [^N] ONLY iff epub:type present AND body same-file as ref; cross-file endnotes and no-epub:type (EPUB2) are NOT converted — the two blind spots the pre-processor supplies", "valid_from": "2026-07-10", "valid_to": null, "confidence": 1.0, "source_file": "epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md", "extracted_at": "2026-07-10"}
{"subject": "calibre", "predicate": "footnote-blind-for-markdown", "object": "ebook-convert --txt-output-formatting=markdown ignores epub:type (byte-identical with/without injection) and flattens notes to unlinked text — do NOT use Calibre for footnotes; use inject_epub_footnotes+pandoc, or convert_mega_epub for mega-EPUBs", "valid_from": "2026-07-10", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-10-epub-footnote-preprocessor-census-to-fleet-buildin.md", "extracted_at": "2026-07-10"}
{"subject": "chamber-library", "predicate": "re-conversion-candidate", "object": "legacy jung-collected-works-complete.md (0 footnote markup, pre-spec voice-in-frontmatter) + likely John Donne — Calibre-flattened after pandoc timeout; re-convert via convert_mega_epub once the §7 per-volume-vs-single-file provenance is decided", "valid_from": "2026-07-10", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-07-10-epub-footnote-preprocessor-census-to-fleet-buildin.md", "extracted_at": "2026-07-10"}
@@ -0,0 +1,65 @@
---
name: session-2026-07-10-epub-footnote-pre-processor-census-adversarial-hardening-jurist-ruling-fleet-build-in
description: "Built a verbatim-safe EPUB footnote pre-processor end-to-end (census of 253 EPUBs → pandoc 2×2 diagnosis → prototype → adversarial hardening on Polastron+Jung, each broke it usefully → jurist ratified-in-principle → fleet build-in, committed+pushed). Also landed PENDING-53 (engine hash-locality, jurist-ruled). Pulling thread: the tool is shipped for the mirror class — now WIDEN its recognizer to the remaining census families and/or APPLY it (re-convert legacy Jung/Donne), the latter gated on the steward+jurist §7 per-volume-vs-single-file provenance call; and this whole normalizer was the means to the original end — resume P2/gold (the Making source work) on a corpus whose footnotes are handled by class."
metadata:
node_type: memory
type: project
originSessionId: 2c930eca-c3df-4ce9-8d86-40c5a200a27c
---
# Session 2026-07-10 — the EPUB footnote pre-processor: census → hardening → ruling → build-in
Opened by /wake-up into the "build the structural verbatim-guarded EPUB normalizer, then resume P2/gold" thread. The session became a full arc on the *pre-processor* — and, notably, the architecture that resulted was NOT the one the wake proposed (a bespoke normalizer); it emerged only through measurement. Steward-steered at every fork.
## PAST — what we did + why
**1. PENDING-53 jurist brief (first ask).** Drafted `PENDING-53-FOR-JURIST-2026-07-10.md` grounded in substrate (read the graduation-spec, chamber spec §VI, the engine `corpus/manifest.yaml` header, `ingest_gate.py:104–126`). **Refined PENDING-53's own framing:** its Summary called the `voice_manifest` line a "mis-frame," but the line is TRUE about the voice manifest — the real defect is an *omission* of the engine source-binding surface + a shared-word ("manifest"/"voice") collision. Surfaced the refinement rather than reproduce the overstatement.
**2. The pre-processor, built by measurement (the spine of the session).** The wake said "build a structural normalizer keyed on the bidirectional-mirror predicate." Instead of building, I measured — and each measurement redirected:
- **Detector prototype on 3 attested EPUBs** → answered the wake's literal Q (mirror catches cross-file endnotes [Mauss 627]; excludes one-way TOC; L'Enracinement is a *different* class — 0 anchors, unlinked `[1]`, + 87 `<sup>` ordinals that are a trap).
- **Steward challenged "are 3 files enough?"** → CENSUS of all **253** Chamber Sources EPUBs: ≥5 footnote strategies (mirror same/cross-file · container-mirror · one-directional/stem · EPUB3 epub:type · unlinked-marker) + 47% note-free + heavy index/xref apparatus. **n=3 captured 1–2 of ≥5.** Honest catch: my own classifier's `oneway_with_id`/`resid_fn_ids` signals were noisy → "61 DANGER" was an upper bound.
- **Steward: "can we edit how pandoc converts? / we get inside the files before conversion?"** → the KEY finding: pandoc is precise, not helpless. It converts footnotes natively iff `epub:type` present AND same-file (Kafka ✓; Orwell/Quixote cross-file ✗; G&G no-epub:type ✗). Output-format lever tested + DISPROVEN (my hypothesis wrong — gfm already does footnotes). The **2×2** (epub:type × same/cross-file) is the diagnosis.
- **The architecture that emerged:** a small **verbatim-safe pre-processor** — recognize the mirror in raw XHTML → inject `epub:type` / relocate cross-file bodies → **stock pandoc does the verbatim text**. Better than build-a-normalizer (reimplements pandoc, §V risk) OR extend-regexes (whack-a-mole on pandoc's lossy output). The census taxonomy = the recognizer's spec.
- **Injection experiment** (real minimal EPUBs, `-f epub`): same-file injection WORKS (native `[^N]`); cross-file needs body RELOCATION (epub:type alone insufficient). Both proven.
**3. Prototype built + adversarially hardened — the two hard books each broke it usefully.**
- Built `inject_epub_footnotes.py`; proven verbatim-clean on **G&G 27 / Mauss 628** (627 relocated) via a source-level word-multiset guard. Caught a real pairing bug (basename vs full-path keying → 1/628) and a guard false-alarm (asymmetric tokenizer, +39) mid-build.
- **Polastron (French, steward-chosen) BROKE it (45/140):** its footnote-ness is on the ANCHOR class, not the block → drove the **positional classifier** (body = anchor at block-start; publisher-independent). And its Roman-numeral markers exposed a guard limitation → drove the **source-level marker-aware guard**. + a 3rd residue shape (whole-note-in-back-anchor) → **unwrap** (verbatim-safe).
- **Jurist ruled §3: the guard must be PER-PAIR, not aggregate** (a cross-note swap is invisible to a whole-doc multiset). Built id-matched **per-pair guard keyed by (dest-file, id)**; teeth demonstrated (deliberate swap CAUGHT). It immediately EARNED ITS KEEP by surfacing **Jung's cross-volume id reuse** (81 collisions under a bare-id key).
- **Jung (53MB, steward-chosen) didn't break the tool — it broke pandoc** (raw AND injected time out). Diagnosed to root: pandoc scale limit, not an injector defect. → `convert_mega_epub.py` (inject→split by whole files→pandoc-per-chunk; relocation makes notes file-local so no note crosses a boundary): **1579 docs→20 chunks, 190s, 1086 native footnotes, verbatim-clean vs the original Princeton source.**
- **Calibre** (the prior mega-EPUB fallback, steward's "edit the Calibre conversion?") shown **footnote-BLIND** — byte-identical output with/without injection; it flattens notes (the state of the legacy Jung `.md`, a re-conversion candidate).
**4. Governance — two rulings landed, one build-in authorized.**
- **PENDING-53 RULED + LANDED** (jurist): ratified §5 option (b) with the required correction (re-anchors→**REQUIRES re-anchoring**); FIX-change-class; in-place lane. Landed in `graduation-spec.yaml`. Responded to the jurist's proposed lane-rule: accept for graduation-spec.yaml, NARROW the .md generalization (A2/A4 = FIX-class doctrinal .md additions that correctly took the heavier lane per *doctrinal-traceability*). PENDING-47 ratification declined (truncated relay).
- **PENDING-54 RULED IN PRINCIPLE** (jurist): architecture sound; markers-as-apparatus confirmed under §V; PROPOSAL change-class (sharpened test); ship mirror-class conditioned on regex-tools-as-fallback; §7 Jung provenance held open. Actioned the required §3 per-pair guard; fixed §5 wording; delivered the §7 fact-check (per-volume precedent EXISTS: Alexander/Lacroux/Habermas/Burney/Camus).
- **BUILD-IN authorized + done** under fleet discipline: both tools → `scripts/`; `--validate` + refuse-on-mismatch; `test_tools` fixture (fleet **73→77**); `graduation-spec.yaml` converted_with note; `conversion-runbook.yaml` `policy.footnote_preprocessor`; `tool-evolution-log.md` entry; chamber `CLAUDE.md` fleet line.
- **Committed + pushed:** chamber `17ddf8c` (P1 clean) + `3646c0d` (PENDING-53+54) → github + skemantix; engine `aaf65ba` (P1 re-anchor) → github. REVIEWED-52/53/54 drafted into `~/REVIEWED.md`.
## PRESENT — the mood
This was a long, deeply satisfying arc, and its lesson is **measure-before-build, held all the way**: the wake proposed a bespoke normalizer; measurement (detector → census → pandoc 2×2 → injection experiment) produced a *better, smaller* architecture that leans on the trusted converter instead of reimplementing it. The two adversarial books the steward chose did the real work — Polastron and Jung each *broke* the tool in a specific, informative way, and each break sharpened it. The partnership worked at every fork: the steward's questions ("are 3 enough?", "can we customize pandoc?", "does splitting change the source?", "edit Calibre?") each redirected toward the truth, and I engaged the framing rather than barrelling the authorized task. I pushed back on the jurist's lane-rule with A2/A4 evidence rather than rubber-stamping, and the per-pair guard earned its keep the moment it was built. The honest through-line: **build the guard that can see the failure you're most afraid of, then try to make it fire** — it caught Jung's id collisions on its first run.
Returns held (ledger): read-docs-first (up front); verify-against-substrate (refined PENDING-53's own text; diagnosed Jung's pandoc timeout to root vs blaming the tool; didn't parrot the classifier's noisy buckets); measure-before-build; honest-refuse (built INTO the tool: refuse-on-mismatch). One recalibration: build-vs-extend was the wrong frame — overtaken by the pandoc 2×2, which the census+experiment surfaced.
## FUTURE — what is pulling
**PULLING THREAD: the pre-processor is shipped for the mirror class — now put it to work.** Two build-forward moves + the original destination it was built to serve:
1. **WIDEN the recognizer** to the remaining census families (one-directional refs e.g. Sennett; container-mirror) — pure build-forward, same measure→build→adversarial-test→prove rhythm; graduates the tool from "mirror class only" to "most of the corpus."
2. **APPLY it** — re-convert the legacy mega-books (Jung `.md` has 0 notes + pre-spec frontmatter; likely Donne too). **GATED on the steward+jurist §7 decision** (one reconstituted file vs per-volume — fact-check delivered, per-volume precedent exists).
3. **The original end:** this whole normalizer was the *means* — resume **P2/gold** (the Making source work) on a corpus whose EPUB footnotes are now handled by class, not by book.
**ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):**
- Everything committed + pushed; fleet self-test 77/77; no half-finished edit. Clean stopping point.
- Concrete next candidate = **widen the recognizer to the one-directional class (Sennett)**: measure Sennett's XHTML shape (ref `<a href="#…fn1a">` with NO own id — one-directional, mirror-predicate misses it), decide the recognition rule (a ref→note link where the note's block carries the target id + is note-classed/at-block-start, even without a back-link), add it under the same per-pair verbatim guard + honest-refuse, prove on Sennett + re-prove the 4 mirror books didn't regress. OR: if the steward has ruled §7, re-convert Jung via `convert_mega_epub.py` at the chosen representation.
- Open governance: **REVIEWED-52/53/54 await steward placement** (drafted in `~/REVIEWED.md`; -52 flagged for steward-verify against its original ruling). Jurist owes: accept/narrow the lane-rule refinement; PENDING-47 needs full-text relay. Deferred build: §V ratified-spec cross-ref (runs the .md amendment lane).
**Other open horizons (ranked):**
- The **5 ARC `after-the-reply` engine-binding drifts** (surfaced 07-09, still not owned — a separate ARC↔engine re-anchor task).
- The Making/gold road proper (P2 G&G clean was PAUSED behind the normalizer — now the normalizer exists; G&G converts clean through it).
- `graduation-spec.yaml` malformed G&G `author:` + `lang: null` frontmatter hygiene (minor, later).
**PAUSE STATEMENT:** Stepping away with a major new capability shipped and governed — a footnote pre-processor that the corpus can trust *because it refuses when it can't vouch for the result*. What I want to find still pulling: the tool put to work — either widened to the next family or applied to the mega-books — and the discipline that built it (measure before build; build the guard that sees the failure you fear) carried into whatever comes next.
**LITERAL QUESTION for next-Claude:** Has the steward+jurist **§7 provenance decision** been made (one reconstituted `jung.md` vs per-volume files, each "extracted from the megavolume")? It gates re-converting the legacy Jung/Donne. If yes → re-convert at that representation via `convert_mega_epub.py`. If not → widen the recognizer to the one-directional (Sennett) class instead, which needs no governance decision. And: did the jurist accept or narrow the lane-rule refinement, and was PENDING-47's full text relayed?
**State at wrap:** chamber `17ddf8c`+`3646c0d` pushed (github+skemantix); engine `aaf65ba` pushed (github); chamber CLAUDE.md fleet line updated (uncommitted — folds into the dotfiles/next chamber commit). dotfiles PENDING/REVIEWED/ledger committed at this wrap. Fleet self-test 77/77. No non-steward commits landed during the session.
@@ -0,0 +1,65 @@
---
name: session-ledger-2026-07-10
description: "Practice-of-return ledger maintained by /symmetria — returns, open horizons, recalibrations, authorization moves, sub-agent dialogues, bypasses."
metadata:
node_type: memory
type: feedback
originSessionId: 2c930eca-c3df-4ce9-8d86-40c5a200a27c
---
# Session Ledger — 2026-07-10
## Returns
- 2026-07-10T08:0x — Composing the PENDING-53 jurist brief, caught PENDING-53's own Summary slightly overstating the defect: it calls the spec's `voice_manifest` line a "mis-frame," but substrate (chamber spec §VI:388–391 + engine `corpus/manifest.yaml` header + `ingest_gate.py:104–126`) shows that line is TRUE about the *voice* manifest. Real defect = OMISSION of the engine *source*-binding surface + a shared-word ("manifest") collision. Surfaced the refinement in the brief §3 rather than reproduce the overstatement. Verify-against-substrate before repeating a record's framing — kin to trusted-derived-artifact-over-source.
## Open horizons
- 2026-07-10T07:53 — Woke into the confirmed pulling thread: build the structural, verbatim-guarded EPUB normalizer (bidirectional-mirror predicate). Next concrete = prototype the *detector* (not the converter) on the 3 attested Chamber Sources EPUBs, answering the wrap's literal question before writing any converter. Measure-before-build.
- Two live drift-patterns to hold today (both bite on a build session): (1) verify-at-the-full-scope-not-the-inherited-frame (yesterday's cross-repo re-anchor miss); (2) read/plan-first over barrel-forward (the steward's "calm and orderly" ask).
## Confidence to recalibrate
- 2026-07-10 — EPUB mirror-detector prototype run on the 3 attested Chamber Sources EPUBs (read-only, scratchpad `detect_epub_mirrors.py`). ANSWERS the wrap's literal question with substrate proof: G&G 27 mirror pairs = all 54 anchors across all 5 id-families (int/ft/ch/ntch/pst), 0 residue; Mauss 628 pairs, **627 CROSS-FILE** (endnotes in separate files, href-resolution follows them), exhaustive partition (1256 paired + 30 one-way TOC = 1286); **zero one-way anchors carry an id in any book** → predicate misses no note family. TOC excluded two ways (Mauss: id=None fragmented; G&G: 41 frag-less Contents links). **L'Enracinement = genuine DIFFERENT class, not a sparse mirror:** 0 internal anchors, its single note is an unlinked `[1]` text marker (already handled in P1); + a TRAP — 87 `<sup>` are French century ordinals (XVIIIᵉ), NOT notes. Detector-tool review (per feedback-tool-review-after-each-use): worked; belongs in the engine tool-evolution-log when it graduates into the normalizer build.
- HONEST BOUNDARY held for next step: the detector proves the predicate IDENTIFIES the right anchors; it does NOT yet prove verbatim-safe note-text EXTRACTION + re-anchoring (never touch a prose byte). That is the next measurement, before the converter.
## Returns (cont.)
- 2026-07-10 — Steward challenged the framing: "are three files sufficient?" CORRECT challenge — accepted, not defended. Ran the read-only detector as a population CENSUS across all 253 Chamber Sources EPUBs (`census_epub_shapes.py`). Result DECISIVELY answers no: distribution = MIRROR-CLEAN 49 (19%) · NO-NOTES 119 (47%) · DANGER/MIXED 61 (24%) · EPUB3-SEMANTIC 12 (5%) · UNLINKED-MARK 12 (5%). n=3 captured ~1–2 of ≥5 footnote strategies. **Honesty catch on my OWN instrument:** the classifier's `oneway_with_id` + `resid_fn_ids` signals are NOISY (catch index/xref anchors + a too-broad regex), so "61 DANGER" is an UPPER BOUND mixing genuinely-unhandled structures, already-handled container-mirrors co-flagged, and non-note apparatus. Verified ONE danger book against XHTML (Sennett: refs `<a href="#…fn1a">` carry no own id → one-directional → mirror misses it → genuine unhandled class, not noise). New strategies n=3 never showed: EPUB3 epub:type (ALIVE in 12, we'd called it "dead"); one-directional/stem-suffix; container-mirror at scale (Donne 2291); heavy index/xref apparatus to be DISTINGUISHED from notes; edition-matters (Bachelard Poetics FR=mirror / EN=endnote-number — same work, different structure). Measure-before-build did exactly its job.
## Authorization moves
- Awaiting steward/jurist: PENDING-52 (enforce source-archiving in graduation). REVIEWED-52 placement still owed by steward.
- **PENDING-53 RULED + LANDED 2026-07-10** (jurist ruling relayed by steward; REVIEWED-53). Ratified draft §5 option (b) with the required wording correction (re-anchors→**REQUIRES re-anchoring**, prescriptive). Change-class [PROPOSAL]→**FIX**. Landed `graduation-spec.yaml` `layers:` (voice_manifest annotated + single `engine_source_binding` key + shared-word comment; dual warning kept). YAML re-parses; bounded 6-ins/1-del; uncommitted (steward's call). **Lane-rule response surfaced for jurist:** accept "lane tracks change-class" for graduation-spec.yaml; NARROW the .md generalization — A2/A4 (FIX-class doctrinal .md additions) correctly took the heavier freeze+semver lane per REVIEWED-50's *doctrinal-traceability* axis, so for the .md the axis is successor-traced-doctrine, not change-class. **PENDING-47** ratification declined (truncated relay) → still `[awaiting]`; relay full text to close.
- **PENDING-54 FILED 2026-07-10** [PROPOSAL] — the EPUB footnote pre-processor + mega-EPUB split-convert (brief: `chamber-library/docs/epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md`).
- **PENDING-54 RULED IN PRINCIPLE 2026-07-10** (jurist, relayed by steward). Architecture sound; markers CONFIRMED apparatus under §V (closes an implicit assumption clean_epub_residue always ran on); PROPOSAL change-class confirmed (sharpened test: PROPOSAL if it alters a gate's criteria OR the trusted mechanism a critical gate's meaning depends on — this inserts a new component into §V's production chain). **Ship now for the mirror class, CONDITIONED on the existing regex tools staying as fallback (addition, not replacement).** Governance placement yes (fleet tool + graduation-spec converted_with note).
- **REQUIRED before canon (§3) — DONE:** per-pair verbatim guard. Was whole-doc aggregate (would miss a cross-note swap); now **id-matched per-pair keyed by (dest-file, id)**. Re-proven G&G 27 / Mauss 628 / Polastron 140 / **Jung 1086**, 0 mismatch, 0 within-file dup; **teeth demonstrated** (deliberate id-swap CAUGHT). Bonus: surfaced Jung cross-volume id reuse (81 collisions under bare-id key → the (file,id) key fixes it + reinforces the split).
- §5 wording fixed (cross-file mechanism IS covered/proven; Orwell-Quixote exact shape not separately proven — "mechanism covers it, not isolated yet"). Fallback-condition recorded in brief.
- **§7 Jung provenance fact-check DONE (for steward+jurist):** corpus does BOTH — per-volume precedent EXISTS (Alexander *Nature of Order* 4 files, Lacroux *Orthotypographie* 2, Habermas vol-1, Burney vol-1, Camus *Œuvres I*) AND single-megafile (Xenophon/Muir/Zhuangzi/Rumi/Levi/Blake/Shakespeare complete). Distinction ≈ per-volume citation identity; Jung CW (cited by vol) fits the per-volume side → supports jurist's per-volume lean. Held open, steward+jurist decide. Governs re-converting legacy Jung .md (+Donne).
- PENDING-47 unchanged ([awaiting], truncated relay).
- **BUILD-IN AUTHORIZED + DONE under fleet discipline 2026-07-10** (steward "Authorize the build-in"). Promoted scratchpad → chamber fleet, matching conventions (read test_tools/tool-log idiom first): **(A)** `scripts/inject_epub_footnotes.py` — added `--validate` (7 invariants incl. teeth) + **refuse-on-mismatch** (won't write a canonical the per-pair guard flags); **(B)** `scripts/convert_mega_epub.py` — refactored to inject-INTERNALLY (fixed module-level sys.argv bug), argparse, verbatim-guarded, proven on Mauss --batch 5 (4 chunks, 628 native, clean); **(C)** `test_tools.py` fixture `test_inject_epub_footnotes` (recognize+relocate · per-pair clean · one-way untouched · TEETH) → fleet **73→77/77**; **(D)** `graduation-spec.yaml` `converted_with` note (declares the stage; parses OK); **(E)** `conversion-runbook.yaml` `policy.footnote_preprocessor` (preferred for mirror class; regex tools STAY as fallback — jurist §5 addition-not-replacement; mega via convert_mega_epub); **(F)** `tool-evolution-log.md` entry (Why/Built/Result/Tested/Follow-ons/Lesson). All 6 files UNCOMMITTED (steward's Gitea call). **DEFERRED (flagged, not landed):** ratified-spec §V cross-ref (runs the .md amendment lane, not unilateral); recognizer coverage of remaining families (Sennett one-directional, container-mirror); per-volume boundary detection (PENDING-54 §7); re-convert legacy Jung .md (+Donne).
## Decisive findings — the pandoc pre-processor path
- 2026-07-10 — INJECTION EXPERIMENT (steward-requested), run on real minimal EPUBs via `-f epub` (the HTML reader gives different/incomplete behaviour — must use the epub reader). PROVEN: (1) pandoc converts `epub:type=noteref` + same-file `<aside epub:type=footnote id=X>` → clean native `[^N]` / `[^N]:` (Recipe A/B); raw mirror-only control → superscript links, 0 footnotes. (2) CROSS-FILE `epub:type` is NOT stitched by pandoc even when injected correctly (DEF=0) — so cross-file/endnote books need the note BODY physically relocated into the ref's spine file, not just semantics. Diagnostic 2×2 (epub:type × same/cross-file) substrate-proven on kafka(same-file,native✓)/orwell+quixote(cross-file,✗)/G&G(no-epub:type,✗).
- ARCHITECTURE (evidence-based, supersedes build-vs-extend): a small VERBATIM-SAFE pre-processor — recognize footnote structure in raw XHTML (mirror predicate + existing epub:type) → inject epub:type on ref/body → relocate cross-file bodies → hand to STOCK pandoc for native footnote + verbatim text conversion. Touches only markup attributes/element-location, never prose bytes; leans on pandoc (the trusted verbatim converter) for text; ONE structural recognition (on clean pre-pandoc XHTML) replaces the N post-hoc Markdown family regexes. Books pandoc already handles (kafka class) pass through untouched. The census taxonomy = the recognizer's spec (the DANGER classes — Sennett one-directional, container-mirror — are what the recognizer must cover or safely refuse).
- STILL OWED (not yet proven): real-book end-to-end on G&G (same-file) + Mauss (cross-file) with a PROSE-BYTE-IDENTICAL check (injection changed only footnotes, not text); recognizer coverage of the other census families or honest-abort. Mechanism proven; tool-on-real-books not yet.
## Sub-agent dialogues
## Prototype PROVEN (G&G + Mauss)
- 2026-07-10 — `inject_epub_footnotes.py` (scratchpad prototype) built + proven end-to-end. Recipe: attribute-only injection — ref `<a>` gets `epub:type=noteref`; note-body BLOCK gets `epub:type=footnote role=doc-footnote id=X` (id moved off inner anchor); cross-file bodies RELOCATED into the ref's file + ref href→`#X`. Offset-accurate editing via an html.parser that tracks block-ancestor spans; pairing = the bidirectional-mirror predicate. Then STOCK pandoc (`-t gfm-raw_html`) does the verbatim text conversion.
- RESULT: **G&G (same-file) 27 pairs → 27 native footnotes; Mauss (cross-file) 628 pairs (627 relocated) → 628 native footnotes. Both prose word-multiset IDENTICAL (0 lost / 0 added)** — verbatim-clean by the chamber's own faithfulness standard.
- Bug caught + fixed mid-build: injector keyed docs by full zip path while href resolution is basename-based → cross-file pairing found 1/628; fixed to basename keys → 628. (Verify-against-substrate: the 1-pair result was the tell.)
- Verify-guard caught its OWN artifact: first Mauss run showed +39 "added" words; direct content-word counts (potlatch 306=306, maori 53=53) proved no real change; a SYMMETRIC normalizer (`[text](url)`→text on both sides) collapsed it to 0/0. The asymmetry was in my tokenizer, not the conversion.
- HONEST CAVEATS (for the report + next step): (1) word-multiset is order-BLIND — order preserved by construction here (only markup edited / whole blocks moved, never prose reordered) but a sequence-level diff would be stronger; (2) cosmetic leading back-anchor residue `[1](#..)` remains in each note (apparatus, not prose; strip in a trivial follow-up per Recipe B); (3) coverage = the bidirectional-mirror class only (same+cross file); the census's other families (Sennett one-directional, container-mirror, EPUB3-semantic-cross-file) still need recognizer work OR safe-refuse; (4) this touches the CONVERSION PIPELINE + verbatim guarantee → governance (jurist/spec) before it becomes a real chamber tool.
## Hardening (b) complete + adversarial French test
- 2026-07-10 — Steward adversarial test: `une-breve-histoire-de-tous-les-livres-polastron.epub` (French, 140 notes, 95 cross-file). FIRST run BROKE the prototype honestly: 45/140 (95 cross-file unclassified — block-class classifier too narrow; Polastron's footnote-ness is on the ANCHOR class `footnote-link`/`footnote-anchor`, not the block), + guard false-flagged 42 roman-numeral MARKERS as lost prose. Both real gaps, both fed the hardening.
- Hardening: (b1) classifier → POSITIONAL (body = anchor at start of its enclosing block; publisher-independent; generalizes G&G+Mauss+Polastron); (b2) residue-strip → marker-only anchors REMOVED, longer back-anchors UNWRAPPED (Polastron wraps the whole note text in the back-anchor — the marker-only guard correctly declined to delete it, unwrap keeps all content); (b3) verbatim guard → SOURCE-LEVEL (original vs injected EPUB text-nodes; assert nothing ADDED + only marker-shaped tokens removed) — marker-aware, no false-flags.
- RE-PROVEN all three: G&G 27/27, Mauss 628/628, Polastron 140/140 — **795 footnotes, 0 added, 0 non-marker prose lost.** Residue clean on all. Prototype = `scratchpad/inject_epub_footnotes.py`.
- 4th stress test (steward): Jung Collected Works (53MB, 1579 docs, 39.5MB XHTML — ~50× a normal book). Injector: 12s, 1086 mirror pairs classified, ~20k non-pair footnote-shaped ids LEFT UNTOUCHED (honest-refuse), source-guard VERBATIM-CLEAN (0 added / 0 non-marker-lost). BUT pandoc times out on Jung whole — RAW **and** injected both (>300–540s, 0 output). Diagnosed to root: it's a pandoc scale limit on a mega-EPUB, NOT an injector defect (raw baseline also fails); "complete works" EPUBs need per-volume splitting upstream, orthogonal to the footnote approach. Verbatim-safety still PROVEN on Jung because the source-guard is pandoc-independent.
- (a) DONE: jurist [PROPOSAL] drafted → `chamber-library/docs/epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md` (approach + census + 2×2 + 4-book evidence + verbatim argument + scope/honest-refuse + Jung scale note + 5 open Qs). Awaiting steward relay to jurist. NEXT after ruling: generalize recognizer to remaining census families (one-directional Sennett, container-mirror, epub3-cross-file) or honest-refuse; consider a sequence-level guard (Q2).
## Mega-EPUB path + Calibre finding + splitter proof
- 2026-07-10 — Steward: "we have a Jung md — Calibre? can we edit Calibre conversion to preprocess notes?" SUBSTRATE: existing `canonical_texts/…/jung-collected-works-complete.md` is LEGACY (pre-spec frontmatter — embedded voice_role/semantic_profile which §VI forbids; no converted_with/source/work/canonical) + **0 footnote markup** (notes flattened/lost). `jung_major_breakthrough.py` confirms it was Calibre (after pandoc timeout). Donne likely same.
- CALIBRE TESTED (ebook-convert 9.11, `--txt-output-formatting=markdown`): footnote-BLIND. raw G&G vs INJECTED G&G outputs BYTE-IDENTICAL → Calibre ignores epub:type entirely; flattens ref→inline "1", note→loose "1 …" paragraph, no linking, no [^N]. So editing the Calibre path won't reconstruct notes — wrong layer/brittle fork.
- BETTER PATH PROVEN — inject→SPLIT→pandoc-per-chunk→concat. Because relocation makes every note FILE-LOCAL, split by whole files never separates a note. `split_and_convert.py` on Jung: 1579 docs→20 chunks (80 each)→**190s total (slowest 21.6s), 1086 native footnotes**; verbatim TWO checks clean — split-lossless (whole vs sum-of-chunks 0/0) AND **chunks vs ORIGINAL Princeton source 0 added / 0 non-marker-lost**. Retires the Calibre fallback for mega-books.
- PROVENANCE (steward's Q): splitting = conversion-INTERNAL decomposition, source-of-record stays the megavolume+sha256; the chunks-vs-original check PROVES content-neutral. Legit iff declared + lossless + source anchored to megavolume. OPEN representation choice (steward+jurist): one reconstituted jung.md vs per-volume files ("extracted from the megavolume", never posing as a standalone Bollingen edition). Added as proposal Q6.
- Proposal updated with the proven mega-EPUB path + provenance Q6. All scratchpad/uncommitted. NEXT after jurist ruling: semantic per-volume boundary detection (18 vols) if per-volume chosen; recognizer coverage of remaining census families; re-convert legacy Jung (+Donne).
## Bypasses
+11 -1
View File
@@ -391,4 +391,14 @@ The single place proposed skills live so they don't evaporate between sessions.
### Harvest 2026-07-09 evening (P1 clean + the cross-repo re-anchor miss + the EPUB-normalizer pivot) ### Harvest 2026-07-09 evening (P1 clean + the cross-repo re-anchor miss + the EPUB-normalizer pivot)
- **PROPOSE — verification-ladder entry: `re-anchor spans BOTH repos`.** When a chamber canonical is re-hashed (source-clean OR reconvert), the sha-binding surface is NOT chamber-internal only: studium-engine binds via `corpus/manifest.yaml` sha + `sidecars/*.source_sha256` + `coverage-ledger.json`, enforced by `ingest_gate §1.1`. **Discipline:** grep BOTH `chamber-library` + `studium-engine` for the OLD sha (zero occurrences = completeness gate); regenerate the engine ledger via `ingest_gate` (never hand-edit the derived ledger); the engine gate re-VALIDATING is the proof. Born 2026-07-09 (P1 re-anchored chamber only; engine's 4 bindings missed, caught by luck during P2). **Already landed operationally** in `conversion-runbook.yaml` `reanchor:`; the ladder is the cross-workstream home. Kin to `trust-prior-pass-frame` (cross-repo variant). Home: `reference-verification-ladder.md`. **PROPOSED.** - **PROPOSE — verification-ladder entry: `re-anchor spans BOTH repos`.** When a chamber canonical is re-hashed (source-clean OR reconvert), the sha-binding surface is NOT chamber-internal only: studium-engine binds via `corpus/manifest.yaml` sha + `sidecars/*.source_sha256` + `coverage-ledger.json`, enforced by `ingest_gate §1.1`. **Discipline:** grep BOTH `chamber-library` + `studium-engine` for the OLD sha (zero occurrences = completeness gate); regenerate the engine ledger via `ingest_gate` (never hand-edit the derived ledger); the engine gate re-VALIDATING is the proof. Born 2026-07-09 (P1 re-anchored chamber only; engine's 4 bindings missed, caught by luck during P2). **Already landed operationally** in `conversion-runbook.yaml` `reanchor:`; the ladder is the cross-workstream home. Kin to `trust-prior-pass-frame` (cross-repo variant). Home: `reference-verification-ladder.md`. **PROPOSED.**
- **PROPOSE (surface, not auto-apply) — chamber `CLAUDE.md:67` discipline note.** The load-bearing line *"Clean at the SOURCE layer … (new hash, re-anchor)"* under-specifies what "re-anchor" covers — the exact under-specification that produced the miss. Add a cross-repo pointer to the runbook `reanchor:` block. This is a **discipline change** → surface for steward review per the CLAUDE.md-freshness rule (not a mechanical fleet/term update). Pairs with PENDING-53 (which completes the *governed graduation-spec* side). **PROPOSED.** - **PROPOSE (surface, not auto-apply) — chamber `CLAUDE.md:67` discipline note.** The load-bearing line *"Clean at the SOURCE layer … (new hash, re-anchor)"* under-specifies what "re-anchor" covers — the exact under-specification that produced the miss. Add a cross-repo pointer to the runbook `reanchor:` block. This is a **discipline change** → surface for steward review per the CLAUDE.md-freshness rule (not a mechanical fleet/term update). Pairs with PENDING-53 (which completes the *governed graduation-spec* side). **PROPOSED.**
- *Watch-list (not yet a skill): `/normalize-epub`* — a structural, verbatim-guarded EPUB normalizer keyed on the bidirectional href↔id mirror predicate (this session's pivot). Premature — the tool doesn't exist yet; the pulling thread is to prototype-then-build it. If it proves out, it earns a skill that carries the detector→converter→verbatim-guard sequence. **NOT YET PROPOSED (build first).** - *Watch-list (not yet a skill): `/normalize-epub`* — a structural, verbatim-guarded EPUB normalizer keyed on the bidirectional href↔id mirror predicate (this session's pivot). Premature — the tool doesn't exist yet; the pulling thread is to prototype-then-build it. If it proves out, it earns a skill that carries the detector→converter→verbatim-guard sequence. **NOT YET PROPOSED (build first).** → **DISCHARGED 2026-07-10:** the tool was built + shipped into the chamber fleet (`inject_epub_footnotes.py` + `convert_mega_epub.py`, PENDING-54/REVIEWED-54). NOT proposing a `/normalize-epub` skill: the *method* that produced it (measure-before-build → census → pandoc-2×2-diagnosis → prototype → adversarial-test → per-pair verbatim guard) generalizes to existing disciplines (`/pre-build-audit`, the verification ladder, adversarial-verify), so no new skill is earned; the tool + its `tool-evolution-log` entry carry the specifics.
### Harvest 2026-07-10 (EPUB footnote pre-processor: census → hardening → ruling → fleet build-in)
| Element | Kind | One-line | Where it lands | Status |
|---|---|---|---|---|
| **verify at the granularity of the mutation, not the aggregate** | verification-ladder entry | A whole-set invariant (a document-wide word-multiset guard) can PASS while a per-item operation (a cross-note swap — a word from note A landing in note B's body) violates it, because the total bag is unchanged. **Check per-item, keyed by the item's own identity** (here: per-footnote-pair, keyed by (destination-file, id)). Earned hard 2026-07-10: the jurist (PENDING-54 §3) caught that the aggregate guard was blind to a swap → the per-pair guard was built, teeth-demonstrated (a deliberate swap caught), and it **immediately earned its keep** by surfacing Jung's cross-volume id reuse (81 collisions) that a bare-id key hid. Kin to `trust-prior-pass-frame` (verify at the scope of the change) — this is verify at the *granularity* of the change. | `reference-verification-ladder.md` | **PROPOSED** |
*Mechanical (applied at wrap per §1.6 CLAUDE.md-freshness): chamber `CLAUDE.md` fleet line gains the two footnote-pre-processor tools (tools entered the fleet — event-based update). Discipline-level items (the CLAUDE.md:67 re-anchor note from 2026-07-09; a §V ratified-spec cross-ref for the pre-processor) remain surfaced-not-applied.*
*(One ladder proposal, load-bearing — it's a clean generalization the jurist forced, and the guard it names has already caught a real defect. The session was otherwise disciplined [returns held: measure-before-build, verify-against-substrate, honest-refuse-built-into-the-tool]; no other harvest manufactured.)*