session 2026-07-11: footnote recognizer arc — hash-locality ratified (12405a5), split-anchor REVIEWED-54a (a6828ac), census re-audit, generalization REVIEWED-54b pt1 block-id+url-decode (ef0cee0); session file + ledger + KG(+4) + skill-harvest

This commit is contained in:
David F Glidden
2026-07-11 01:22:36 +02:00
parent 9e148b7a8f
commit 5ec438ce72
6 changed files with 113 additions and 1 deletions
@@ -10,6 +10,8 @@ metadata:
# Session Ledger — 2026-07-10
## Returns
- 2026-07-10T17:1x — **Over-reached on an outward action; auto-classifier caught it.** Bundled a `git push` to chamber `main` into the jurist-ruling commit command — but the steward relayed the ruling, never directed a push; a direct default-branch push bypasses PR review + is outward-facing. Correct move (adopted): local commit only (reversible checkpoint `12405a5`), surface the push for steward direction. Barrel-forward on an outward action — the 07-09 "calm and orderly" drift, one notch more specific: commit ≠ push; the review boundary is the push, not the checkpoint.
- 2026-07-10T16:48 — **Post-clear wake.** Caught + disregarded the SessionStart hook's stale 2026-03-05 handoff (BMF/Gitea era); reconstructed from the live memory layer. Thread validity CONFIRMED (pre-processor shipped for mirror class; next = widen to Sennett OR apply to Jung, the latter gated on §7). Symmetria re-anchored. Drift to hold: barrel-forward-vs-read-plan-first (07-09 "calm and orderly") + per-item-granularity over aggregate invariants (07-10 jurist §3).
- 2026-07-10T08:0x — Composing the PENDING-53 jurist brief, caught PENDING-53's own Summary slightly overstating the defect: it calls the spec's `voice_manifest` line a "mis-frame," but substrate (chamber spec §VI:388–391 + engine `corpus/manifest.yaml` header + `ingest_gate.py:104–126`) shows that line is TRUE about the *voice* manifest. Real defect = OMISSION of the engine *source*-binding surface + a shared-word ("manifest") collision. Surfaced the refinement in the brief §3 rather than reproduce the overstatement. Verify-against-substrate before repeating a record's framing — kin to trusted-derived-artifact-over-source.
## Open horizons
@@ -24,6 +26,21 @@ metadata:
- 2026-07-10 — Steward challenged the framing: "are three files sufficient?" CORRECT challenge — accepted, not defended. Ran the read-only detector as a population CENSUS across all 253 Chamber Sources EPUBs (`census_epub_shapes.py`). Result DECISIVELY answers no: distribution = MIRROR-CLEAN 49 (19%) · NO-NOTES 119 (47%) · DANGER/MIXED 61 (24%) · EPUB3-SEMANTIC 12 (5%) · UNLINKED-MARK 12 (5%). n=3 captured ~1–2 of ≥5 footnote strategies. **Honesty catch on my OWN instrument:** the classifier's `oneway_with_id` + `resid_fn_ids` signals are NOISY (catch index/xref anchors + a too-broad regex), so "61 DANGER" is an UPPER BOUND mixing genuinely-unhandled structures, already-handled container-mirrors co-flagged, and non-note apparatus. Verified ONE danger book against XHTML (Sennett: refs `<a href="#…fn1a">` carry no own id → one-directional → mirror misses it → genuine unhandled class, not noise). New strategies n=3 never showed: EPUB3 epub:type (ALIVE in 12, we'd called it "dead"); one-directional/stem-suffix; container-mirror at scale (Donne 2291); heavy index/xref apparatus to be DISTINGUISHED from notes; edition-matters (Bachelard Poetics FR=mirror / EN=endnote-number — same work, different structure). Measure-before-build did exactly its job.
## Authorization moves
- **2026-07-10T16:5x (post-clear) — §7 (PENDING-54 mega-EPUB provenance, Jung) AUTHORIZED by steward.** Per-volume separation, **each volume referencing the parent collection** (source/source_verified → the single megavolume sha256; frontmatter states extraction, never a standalone Bollingen edition). Steward exercised final authority on the point the jurist held OPEN with a per-volume lean → aligns, NO new jurist round. Unblocks re-converting legacy Jung `.md` (+Donne) via `convert_mega_epub.py` at the per-volume representation; open design sub-task = per-volume boundary detection (~18 CW vols). §7 resolution text PRESENTED to steward for placement in REVIEWED-54 — NOT written to REVIEWED.md (Constitutional Constraint #1: executor cannot modify REVIEWED.md; the executor-drafts-marked-entries practice of 52/53/54 sits in tension with it — flagged for steward). Held execution (calm-and-orderly) pending steward go.
- **CORRECTION (substrate-verified):** PENDING-47 (the whole stress-test/source-match/exclusion arc) is CLOSED + ruled (REVIEWED-47, 2026-07-05). My post-clear wake briefing MISLABELED the open jurist item as "PENDING-47" — inherited loosely from this ledger's shorthand + carried forward without verifying (inherited-marker-read-as-current-state). The TWO genuinely-open jurist items (both REVIEWED-53): (a) the **generalized hash-locality principle** — `graduation-spec.yaml` L29–39, marked `[awaiting jurist ratification of the generalized principle]`; the 07-10 relay was TRUNCATED so the jurist declined to ratify unread; fix = relay the full untruncated text; (b) the **lane-rule accept/narrow** (REVIEWED-53 L474). Corrected against the substrate before propagating further.
- **GOVERNANCE STATE (post-clear, Fri 2026-07-10):**
1. Steward working remotely from Ravello. **REVIEWED placements OWED BY STEWARD → deferred to Sunday 2026-07-12** (steward's own commitment): confirm/place REVIEWED-52/53/54 + ADD the §7 block below to REVIEWED-54. **Do NOT re-surface as dropped before 07-12** — it is intentionally held, not lost.
2. **Two jurist items RULED 2026-07-10** (jurist reply relayed by steward; verbatim archived → `chamber-library/docs/hash-locality-ratification-and-lane-narrowing-JURIST-RULING-2026-07-10.md`): **(a)** generalized hash-locality principle **RATIFIED in full** (all three instances checked, not assumed) → executor CLOSED the `graduation-spec.yaml` marker, dated + trued 2→3 instances; committed `12405a5`, **PUSHED to github + origin/Gitea 2026-07-10 on steward direction** (both remotes clean; Gitea cooperated). **(b)** lane doctrine **NARROWED — executor's correction ACCEPTED:** machine-data files keep change-class-tracks-lane (they have a code/git/test_tools fallback); the `.md` constitution runs on **doctrinal-traceability** (its prose is the sole record); axis = doctrinal content, not file extension (a no-doctrine typo/deadlink `.md` fix still doesn't need the heavy lane). **Steward-owed Sun 2026-07-12:** record (b) against REVIEWED-53 §2 (paste-ready below).
3. §7 AUTHORIZED → legacy Jung per-volume re-convert is UNBLOCKED (no further governance needed). Execution HELD pending steward go.
- **§7 paste-ready block for REVIEWED-54 (steward to place Sunday 2026-07-12):**
> **§7 RESOLVED 2026-07-10 (steward, final authority on the point the jurist held open with an aligned per-volume lean):** AUTHORIZED — **per-volume separation, each volume referencing the parent collection.** Jung CW extracted into per-volume files; each carries `source` / `source_verified` → the **single megavolume sha256**; frontmatter states extraction from the collection, **never poses as a standalone Bollingen edition** (the jurist's airtight-safeguard condition, adopted). Matches the jurist's agreed direction → no new jurist round. Unblocks re-converting the legacy Jung `.md` (+Donne) via `convert_mega_epub.py` at the per-volume representation. Open design sub-task: per-volume boundary detection across the ~18 CW volumes (`convert_mega_epub.py` currently splits by 80-doc chunks, not by volume).
- **Lane-doctrine paste-ready for REVIEWED-53 §2 (steward to record Sun 2026-07-12):**
> **Lane-rule refinement — jurist ACCEPTED + narrowed (ruling 2026-07-10).** "Lane tracks change-class" stands for `graduation-spec.yaml` / machine-data files only — they carry an independent fallback (code, git history, `test_tools` fixtures) that already pins "what was true when," so change-class is a sufficient proxy. The `.md` constitutional spec has no such fallback — its prose IS the sole authoritative record — so its lane tracks **doctrinal-traceability**: whether the change adds doctrine a successor must trace to a dated marker, regardless of change-class. NOT "every `.md` edit takes the heavy lane" — a no-doctrine typo/deadlink fix doesn't. The axis is doctrinal content, not file extension. This narrows the PENDING-53 ruling (§2) for anything beyond `graduation-spec.yaml` itself; that document's proposed rule is read with this correction attached.
- **REVIEWED-54a paste-ready (steward to place Sun 2026-07-12; executor cannot write REVIEWED.md — Constraint #1):**
> **## REVIEWED-54a — Split-anchor footnote recognizer widen (Sennett) + GOV-2 dup-scope**
> **Date:** 2026-07-10 · **Decision:** AUTHORIZED (jurist FIX-ruled 2026-07-10) · **extends:** REVIEWED-54
> **Notes:** Recognizer widen to the split-anchor footnote family (first REVIEWED-54 deferred family). Jurist FIX-class via the reusable test — a widen stays FIX iff its safety case is carried entirely by the SAME unchanged verification (a new recognition predicate, not a change to how recognition is verified); crosses to PROPOSAL the moment it modifies the guard, adds a new verification mechanism, or hands prose conversion to anything other than stock pandoc. GOV-2 = option (c): the per-pair guard's within-file dup-check is scoped to conversion-touched ids (note-body / mirror-ref / split back-target — incl. the jurist's back-target precision); an unrelated pre-existing dup is a non-blocking WARNING routed to anchor-hygiene.
> **If AUTHORIZED:** LANDED + PUSHED (`a6828ac`, github+Gitea 2026-07-10) — split-anchor pass + GOV-2-scoped `verify_pairs` in `scripts/inject_epub_footnotes.py`; `test_tools` fixture (fleet 77→80); `--validate` 15; runbook policy + tool-evolution-log + chamber CLAUDE.md. Verbatim-clean on Sennett (294/294 → native, 0 prose added/lost). Jurist ruling + brief in `docs/`. Separate travel-items: the `cra0001002` anchor-hygiene dup-id audit; the census-misclassification flag. Tag follow-ons REVIEWED-54a.
- Awaiting steward/jurist: PENDING-52 (enforce source-archiving in graduation). REVIEWED-52 placement still owed by steward.
- **PENDING-53 RULED + LANDED 2026-07-10** (jurist ruling relayed by steward; REVIEWED-53). Ratified draft §5 option (b) with the required wording correction (re-anchors→**REQUIRES re-anchoring**, prescriptive). Change-class [PROPOSAL]→**FIX**. Landed `graduation-spec.yaml` `layers:` (voice_manifest annotated + single `engine_source_binding` key + shared-word comment; dual warning kept). YAML re-parses; bounded 6-ins/1-del; uncommitted (steward's call). **Lane-rule response surfaced for jurist:** accept "lane tracks change-class" for graduation-spec.yaml; NARROW the .md generalization — A2/A4 (FIX-class doctrinal .md additions) correctly took the heavier freeze+semver lane per REVIEWED-50's *doctrinal-traceability* axis, so for the .md the axis is successor-traced-doctrine, not change-class. **PENDING-47** ratification declined (truncated relay) → still `[awaiting]`; relay full text to close.
- **PENDING-54 FILED 2026-07-10** [PROPOSAL] — the EPUB footnote pre-processor + mega-EPUB split-convert (brief: `chamber-library/docs/epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md`).
@@ -62,4 +79,24 @@ metadata:
- PROVENANCE (steward's Q): splitting = conversion-INTERNAL decomposition, source-of-record stays the megavolume+sha256; the chunks-vs-original check PROVES content-neutral. Legit iff declared + lossless + source anchored to megavolume. OPEN representation choice (steward+jurist): one reconstituted jung.md vs per-volume files ("extracted from the megavolume", never posing as a standalone Bollingen edition). Added as proposal Q6.
- Proposal updated with the proven mega-EPUB path + provenance Q6. All scratchpad/uncommitted. NEXT after jurist ruling: semantic per-volume boundary detection (18 vols) if per-volume chosen; recognizer coverage of remaining census families; re-convert legacy Jung (+Donne).
## Sennett recognizer-widen (autonomous — steward flying, 2026-07-10 pm)
- **SOLVED + PROVEN as a scratch prototype; governed fleet tool UNTOUCHED.** Root cause the tool refused Sennett: the mirror pairing keys on ANCHOR ids and requires the note to link back to the ref anchor's own id; Sennett is a **split-anchor mirror** — note id on the `<p>` BLOCK, back-target on an empty href-less sibling `<a id="…a">` the parser didn't even record → no Sennett anchor entered pairing → honest-refuse.
- **Extension** (`scratch/inject_splitanchor.py`, additive 2nd pass): record href-less id anchors; pair forward-ref → note BLOCK id, reciprocity verified via the sibling back-target anchor CO-LOCATED with the ref (full round-trip, as safe as the direct mirror) + a ref-is-marker-shaped gate; body_id falls back to the block id. Same per-pair guard + refuse-on-mismatch, unchanged.
- **Proof:** `--validate` **12/12** — 7 original invariants INTACT (no regression; additive) + 5 new (incl. honest-refuse negative: a one-way link to a non-back-linking block is left untouched). Real Sennett: **294/294 footnotes, all cross-file, 0 unclassified, 0 prose mismatch**; end-to-end through stock pandoc → **294 native footnote defs, 0 prose added / 0 non-marker prose lost = VERBATIM CLEAN** (`scratch/_sennett_prove.py`). Measurement-tool lesson (NOT the conversion): the pandoc-level word-multiset must strip markdown link TARGETS + treat UNICODE superscripts as markers (same family as the morning's Mauss +39 false-alarm); the conversion was clean throughout, proven independently by the source-level per-pair guard.
- **BLOCKER isolated (honest-refuse fired correctly):** `cra0001002` — a **PRE-EXISTING non-footnote duplicate block id** in `part0003` (front matter), present in the RAW source before injection; NOT a note id, NOT caused by the extension. Both follow-ups HELD for governance:
- **(GOV-1) Build-in authorization:** widening the recognizer to a new family is named in REVIEWED-54's deferred list ("recognizer coverage of remaining families"). Verbatim safety mechanism unchanged (same guard) → likely within that deferral / FIX-class, but steward+jurist to rule vs a fresh beat (jurist §4 test).
- **(GOV-2) Guard-scoping refinement (jurist §3 mechanism):** the per-pair guard's within-file-dup check flags ANY block-id dup, incl. pre-existing NON-note dups unrelated to the conversion → couples footnote-conversion refusal to unrelated source defects. Scope the dup check to the NOTE ids the conversion touches, OR require the source dup-clean first? Surface, don't decide.
- **(source-hygiene) Sennett source carries a pre-existing dup id `cra0001002`** (front matter) — a separate corpus-cleaning item regardless.
- **HELD:** fleet build-in + commit await steward+jurist (GOV-1). Prototype + proof harness in `scratch/`; `scripts/inject_epub_footnotes.py` untouched.
- **JURIST BRIEF DRAFTED** (`docs/split-anchor-recognizer-widen-FOR-JURIST-2026-07-10.md`, UNCOMMITTED for steward review): asks GOV-1 (build-in + change-class; recommends FIX within REVIEWED-54's deferred "recognizer coverage") + GOV-2 (scope §3 dup-check to conversion-touched note ids; recommends option (c): scope + report-not-block for non-note dups). Full untruncated text handed to steward for relay (the PENDING-47 truncation lesson). Awaiting jurist ruling to close.
- **RULED + LANDED (jurist 2026-07-10, relayed by steward; archived `docs/sennett-split-anchor-widen-JURIST-RULING-2026-07-10.md`).** GOV-1 authorized **FIX-class** (jurist stated a reusable test: a widen stays FIX iff its safety case rides the SAME unchanged verification — a new recognition *predicate*, not a change to how recognition is *verified*). GOV-2 = **option (c)** confirmed WITH a required precision: the dup-scope must include split-anchor **back-target ids**, not only primary note ids (implemented + teeth-tested). Build-in DONE under fleet discipline: split-anchor pairing pass + GOV-2-scoped `verify_pairs` in `scripts/inject_epub_footnotes.py`; `test_tools` split-anchor+GOV-2 fixture (fleet **77→80**); `--validate` **15** (7 original INTACT); runbook policy + tool-evolution-log + chamber CLAUDE.md updated. Sennett proven verbatim-clean (294/294 → native; `cra0001002` → WARNING not block). **Committed `a6828ac`, PUSHED to github + origin/Gitea 2026-07-10 on steward direction (both clean).** **Jurist flag carried forward:** the census MIS-FILED Sennett as "one-directional" — its classifier missed the same back-link the recognizer did; before the NEXT family widen, check it isn't a census misclassification of a covered class (cheaper to re-measure than to build).
## Census re-audit (steward-chosen next move; jurist census-misclassification flag)
- **Ran the CURRENT recognizer over all 252 Chamber Sources EPUBs** (`scratch/_reaudit_families.py`, read-only, 75s) — classifies by what the tool NOW does, not the census's blind-spotted classifier. Result: **TOOL-COVERED 65** (59 mirror + **6 split-anchor / 503 footnotes** — 6 books the census would've mis-filed "one-directional", the widen's real ROI) · PANDOC-NATIVE 4 · **UNCOVERED 40** · NO-NOTES 92 · MEGA-SKIP 51 (>4MB; many have notes → convert_mega_epub track) · ERROR 0.
- **Structural probe of 8 high-signal UNCOVERED books** (`scratch/_probe_uncovered.py`): the "40" is NOT 5 families. Breakdown: **(A) ~9–12 false positives** (poetry/nav cross-refs, fn_ref≤3: blake=1, and-our-faces/photocopies/titian/juvenescence/between-you-me/duino=2; plato's 387 is mostly nav page-anchors). **(B) URL-decode GAP, not a family** — la-pensée-sauvage refs are `%20`-encoded (`norm_href` doesn't `unquote`) → cross-file resolution fails; a 1-line fix. **(C) NEAR-MIRROR VARIANTS** (the genuine bulk, ~25–30 books incl. haraway 2613 / genette 1050 / doniger 1057 / mbembe 523 / kuhn 467 / nagarjuna 423 / nature-of-things 354 / bachelard 243 / jacobs 142): a note BLOCK carries the note id and back-links to the ref — differing from the handled split-anchor only in WHERE the id sits (on the `<p>` block, or an inner empty `<a id>` at block-start — kuhn/bachelard) and whether the ref carries its OWN id vs a sibling (haraway/genette/jacobs). **(D) letter-markers** — donne is a split-anchor book refused only because markers are a/b/c (marker gate = digits+roman).
- **CONCLUSION (answers the jurist flag):** the remaining footnote scope is **≈ ONE recognizer generalization + two small fixes (URL-decode, letter-marker), not five families.** The generalization: "note identity may sit on the block OR an inner empty anchor at block-start; ref identity on the ref anchor OR a sibling; reciprocity via any co-located anchor." Likely **PROPOSAL-class** (it broadens the recognition predicate more than split-anchor did — though still riding the same unchanged verification, the jurist's line is whether the *predicate* generalization needs a fresh beat) → surfaced for steward+jurist, NOT built. Alternative: pivot to APPLYING the tool (gold road — Sennett/Jung re-convert). **HELD for steward decision.** Artifacts: `scratch/_reaudit_families.py`, `scratch/_probe_uncovered.py`.
- **QUANTIFIED + JURIST BRIEF PREPARED (2026-07-11, steward relaying to jurist).** Exact UNCOVERED-40 (`scratch/_reaudit2.py`): **14 block-id near-mirror · 6 inner-anchor near-mirror · 2 letter-marker · 3 URL-decode · 7 other(span-wrapped, regex-missed) · 8 false-pos(low)**. Reciprocity VERIFIED exact: haraway ref `<a id="c1_fn1" href="notes#c1_rfn1">` ↔ note `<p id="c1_rfn1">` back-links c1_fn1 (block-id = exact mirror, safer than split-anchor's adjacency); kuhn `<p class=footnote><a id="intfn1"/>…` (inner-anchor). **The "remaining families" = ONE reciprocal mirror with the id in 3 positions + 2 mechanical gaps (letters, %-encoding) — NOT 5 families.** Brief = `docs/recognizer-generalization-FOR-JURIST-2026-07-11.md` (UNCOMMITTED for steward review). **Change-class read (applying the jurist's own REVIEWED-54a test part-by-part, NOT reflexively 'PROPOSAL'):** block-id / inner-anchor / URL-decode = **FIX** (same unchanged verification, exact reciprocity, split-anchor precedent); letter-marker = the ONE part that loosens a guard (condition-4 defense-in-depth) → proposed compensation = require EXACT reciprocity for letter-marked notes; jurist to rule FIX-with-compensation vs PROPOSAL. Offered: prototype-and-prove exact coverage before the ruling (split-anchor pattern). Awaiting jurist ruling.
- **RULED (jurist 2026-07-11, relayed by steward; archived `docs/recognizer-generalization-JURIST-RULING-2026-07-10.md`).** Census-flag verdict CONFIRMED (one family, three id-positions). **(a) block-id, (b) inner-anchor, (d) URL-decode = FIX** (extends REVIEWED-54a; jurist noted (a)/(b) are STRONGER than split-anchor — exact id-match reciprocity, not adjacency). **(c) letter-markers = FIX-with-exact-reciprocity-compensation** (letter-marked notes REQUIRE exact reciprocity, no adjacency fallback) **+ a proportionate condition: human spot-check the 3 named letter-marker books (nagarjuna, plato, donne) before trusting their conversion** (n=3, cheap; not PROPOSAL). **Design ruled NOW; prototype-proven coverage required AT BUILD** (not a precondition), same standard split-anchor met. **Tag = REVIEWED-54b.** Non-blocking asks DONE: (1) degraded-canonicals tracking line → `_curation/reconvert-queue-2026-06-29.md` new section (montaigne/meditations/la-pensée-sauvage/l'homme-t2/donne/jung); (2) 252-vs-253 reconciled (archive holds 252 `.epub`; "253" was the morning's loose count — 252 confirmed). **BUILD PLAN:** implement (a)(b)(d) + (c)-with-compensation + GOV-2 touched-id extension in the prototype → `--validate` invariants per id-position → prototype-prove verbatim-clean on the real UNCOVERED books → port to fleet + `test_tools` fixtures → commit (push held) → surface the 3 letter-marker pairs for steward spot-check → draft REVIEWED-54b.
- **BUILD PROGRESS (prototype `scratch/inject_splitanchor.py`, 2026-07-11):** **(a) block-id + (d) URL-decode DONE + PROVEN verbatim-clean end-to-end on REAL books** (`scratch/_prove.py`): **haraway 322→322 native · genette 350→350 · jacobs 71→71 · la-pensée-sauvage 87 (turned out a plain MIRROR blocked only by %20 — url-decode alone fixed it) → 88 · Sennett 294 regression CLEAN**, all **+0/−0 prose**. `--validate` **18** (15 prior + block-id×2 + url-decode). Implementation: `norm_href` `unquote`; block-note pass now accepts **ref-own-id (exact reciprocity)** alongside the sibling-adjacency split case; **latent DUP-ID FIXED** (strip block id before re-adding epub:type id — was a real defect in the shipped Sennett code, `<p id=X epub:type=… id=X>`; now single id); entity-aware marker detection (haraway's `&#160;&#160;1`). **REMAINING (next focused pass — the fiddlier parts, deliberately NOT rushed at a long session's tail per τὸ πρόσφορον):** (b) inner-anchor (relocate id from inner empty `<a id/>` to the block without a dup the guard would rightly block) + (c) letter-marker (marker-gate + exact-reciprocity compensation), then port ALL of REVIEWED-54b → fleet + `test_tools` fixtures + commit (push held) + surface the 3 letter-marker pairs (nagarjuna/plato/donne) for steward spot-check + draft REVIEWED-54b. Prototype proven; NOT yet ported to the fleet tool.
## Bypasses