dotfiles/claude/memory/session-2026-07-12-A1-to-docling-adoption-and-format-decision.md

name, description, metadata

name	description	metadata
session-2026-07-12-a1-verbatim-gate-steward-reframe-docling-adoption-canonical-format-decision-pending-56-in-flight-sidecar-build	Woke into A1 (the body_word_conservation gate); built it (shingle-coverage, difflib→shingle pivot on scale). Steward challenged the framing TWICE (difflib-doesn't-scale, then no-extant-tools/TEI) → the pivotal reframe: the CONVERSION SUBSTRATE is a solved adoptable domain (Docling/GROBID/TEI/DTS) and we'd spent 6 weeks rebuilding it — the answer was already in our own tooling-register (2026-05-27). Ran a 4-tier Docling trial (M4+M1): cleans EPUB, rich structure+provenance on PDF, CROSSES the A3 OCR frontier on scanned French. Steward DECIDED (AskUserQuestion) + jurist RATIFIED PENDING-56: Option A = MD-canonical + TEI-mirroring .meta.json structural sidecar; adopt Docling as front-end. Jurist closed 2 schema-gating Qs (recognizer-repositioned-not-retired; preserve-never-flatten cardinal principle). In-flight (steward travelling): built the sidecar schema DRAFT + build_sidecar.py extractor (proven on montaigne/jacobs; honest join-limit), updated runbook + both CLAUDE.mds, left an in-flight report. PULLING THREAD: lock the DRAFT schema (Q-A) + confirm the Docling-per-spine-file architecture (Q-B), then build the wired extractor. Steward owes REVIEWED-55/56.	node_type type originSessionId memory project ae9583d2-fd25-45e4-85fc-df082fe2e951

Session 2026-07-12 — A1 → the Docling reframe → the canonical-format decision → in-flight sidecar build

An exceptionally long, generative session that started narrow (build one gate) and, through two sharp steward challenges, turned into the program's biggest architectural pivot yet: stop hand-building a conversion pipeline; adopt the mature substrate (Docling); keep our trust layer; decide the canonical format.

PAST — what we did + why

1. Woke into A1 (the general body_word_conservation verbatim gate — the open-work register's keystone). Read the substrate first and found the register's own spec already RATIFIES the per-tier method (§Tiering&Fence: V-DSL re-extract+diff; V-TEXT re-convert+prose-word-guard; V-SCAN abstain). So A1 was operationalize a ratified criterion, not invent one. Built verify_body_conservation.py — reference-conversion + shingle-coverage alignment with a three-tier verdict (FLAG interior-loss/fabrication · REVIEW boundary-drops-need-id · PASS). Seed ground-truth passed; caught real cruft in 3 "clean" canonicals (jacobs 888 calibre_link, manguel 2856 []{#…}, montaigne fn-ids — all grep-confirmed).

2. Steward challenge #1 (difflib doesn't scale): difflib.SequenceMatcher (O(n²)) timed out on jacobs @2min. Pivot to O(n) hashed-k-gram shingle-coverage (in-house patterns: Instrument-B fingerprint + verbatim-guard walk; NO new dep). jacobs 6.8s. Well-caught by the steward.

3. Steward challenge #2 (are there no extant tools? MD the problem? TEI?): THE PIVOT. Honest finding: the conversion/format/citation SUBSTRATE is a solved, adoptable domain (Docling/GROBID/TEI/DTS) — and the answer was already in our own tooling-register (2026-05-27); we'd spent ~6 weeks hand-building ~20 bespoke tools instead. The two-week "every breakthrough → new obstacle" feeling = signature of reimplementing a solved domain. Ran /tooling-scan studium-engine (web-grounded); verified Docling (MIT, local, EPUB+PDF+images, provenance, layout-derived labels) + GROBID (PDF→TEI) + DTS. Clean split: substrate = adopt; composition (the engine) = ours. Register updated.

4. The Docling trial (steward-run, M4+M1, 4 tiers): value scales with layout richness — flat on EPUB, rich on PDF, transformative on scanned. EPUB: 0 cruft, notes+headings preserved, flat semantics (reflowable = no pages). text-PDF: footnote/header/section labels + 2 798 page-provenanced items. scanned French (A3 frontier ocrmac can't cross): OCR'd, 104 footnotes, 664 provenanced — readable but with OCR defects → feeds normalize+verify, doesn't replace. Docling is a converter, not a verifier. Loeb is a .dsl (already structured) → NOT a Docling case (steward correction).

5. Canonical-format DECISION (PENDING-56). Wrote the brief (chamber-library/docs/canonical-format-decision-FOR-JURIST-2026-07-12.md). Steward decided via AskUserQuestion: Option A (MD-canonical + TEI-mirroring .meta.json structural sidecar) + Adopt Docling. Grounding that clinched A: the conversion-plan already specifies MD+sidecar and the engine already rules D-2 (MD+sidecar, TEI later) + consumes 19 .meta.json sidecars — so A completes a chosen direction, TEI would depart it. Change-class PROPOSAL.

6. Jurist RATIFIED (independently-verified, not deference). Flip-condition tested, NOT triggered for v1 (voice-manifest = interpretive-persona per PENDING-53, not manuscript-variant; Loeb needs anchors [B2, which A provides] not apparatus [A5]). Closed 2 schema-gating questions with substrate evidence (brief §8): Q1 — Docling emits flat [*](#…) not native [^N] (0/48; epub_backend ignores epub:type) → the footnote-recognizer arc is REPOSITIONED as the sidecar note-layer, not retired; Q2 — the Loeb DSL is TYPED ([c dimgray] 2.15M / [c darkmagenta] 471K / [sup]) and the current extractor flattens it → CARDINAL PRINCIPLE: preserve source typing into typed sidecar fields, never flatten (→ TEI-migration mechanical-by-construction). PENDING-56 filed; REVIEWED-56 drafted (steward to file).

7. In-flight autonomous build (steward travelling; read-only/dev-only; 8 tasks): sidecar schema DRAFT (docs/sidecar-schema-DRAFT-v0-2026-07-12.md, extends studium/meta@1); build_sidecar.py v0 — recovers structure from source metadata (epub:type/landmarks → guide/NCX/filenames → honest-flag). Proven: montaigne 16/17 declared, jacobs 37/37 (landmark propagation), provenance 400, notes 56. Honest v0 limits: the Docling content-match line-join is brittle on drop-caps/heading-format divergence (montaigne 2/17, jacobs 34/37) → real fix = Docling per-spine-file (exact ranges by construction) + reuse inject_epub_footnotes for the note-layer. Tool-review logged. Updated conversion-runbook.yaml (machines: M4/M1 split + pipeline_direction_pending_56 + A3 gap), chamber + studium CLAUDE.mds. Report: in-flight-report-2026-07-12.md.

PRESENT — the mood

Two-phase, like the previous session. First: careful A1 engineering — but the steward's two challenges revealed I was solving at the wrong level (building heuristics to recover from lossy MD what a semantic format/tool preserves natively; and hand-building what the ecosystem + our own register already had). That's the executor-bias-to-composition, one level up — and the steward named it precisely ("every breakthrough lands us in front of the next obstacle"). Second phase: clean architectural decision-making — brief → steward decision → jurist ratification → in-flight execution, each link governed. Returns worth carrying: verified before asserting throughout (the difflib timeout, the docling flat-footnotes, the Loeb DSL typing, the montaigne join-bug — all grep/measure, not inference); honest about the v0's limits (didn't dress up the 2/17 join or the per-publisher quirks); respected the loop (drafted the schema, did NOT lock it; surfaced decisions as Q-A..Q-E, didn't answer them). Recalibrations: wrong-level (heuristic over substrate), steward-named; check our own register/notes before building (the resurface-banked-notes drift at pipeline scale — the register had the answer 6 weeks ago).

FUTURE — what is pulling

PULLING THREAD: lock the DRAFT sidecar schema (Q-A) + confirm the Docling-per-spine-file architecture (Q-B) → build the WIRED extractor. The v0 proved the thesis (structure recoverable from source metadata, dissolving the drop-vs-keep heuristic) but the docling line-join needs the per-spine-file architecture, and the schema is curator-locked (steward+jurist), not executor-locked.

ACTIONABLE RESUMPTION POINT (as of wrap — steward said he'll CLEAR + /wake-up fresh):

• Read in-flight-report-2026-07-12.md (the full account + Q-A..Q-E) + docs/chamber-program-open-work.md (forest view) first.
• Q-A (lock schema): review docs/sidecar-schema-DRAFT-v0-2026-07-12.md against the proven extractor output (_curation/sidecar-drafts/) → lock via the amendment process.
• Q-B (architecture): confirm Docling-per-spine-file (exact line-ranges) over the v0 content-match join; then build the wired extractor + reuse inject_epub_footnotes as the note-layer.
• Then: the graduation-spec amendment (step 6, PROPOSAL) wires Docling+sidecar as the graduated structural layer + retires ocrmac_pdf/EPUB cruft-strippers; reshapes reprocess track B.

Other open horizons (ranked): Loeb B2 (the far-country lever; the [c darkmagenta] content question resolves by sampling) · fleet-promotion of build_sidecar.py (into test_tools, after lock) · the engine track (V1/N1) runs parallel, unblocked.

Steward owes (parallel, he can't-be-me): file REVIEWED-55 + REVIEWED-56 (REVIEWED-drafts-for-filing-2026-07-12.md); relay the PENDING-56 §8 closure to the jurist (or note ruled).

PAUSE STATEMENT: Stepping away at a clean, sealed point — the biggest architectural decision of the program made, ratified, and its two open questions closed; the first execution step (schema + extractor) built and proven, with the constitutional pieces (schema-lock, amendment, REVIEWED filing) correctly left for the steward. The steward will clear + wake fresh. What I want to find still pulling: Q-A/Q-B — lock the schema, confirm the architecture, build the wired extractor. All chamber/studium/CM-AI work is UNCOMMITTED working-tree (no-push guardrail); nothing lost.

LITERAL QUESTION for next-Claude: Did the steward lock the DRAFT sidecar schema (sidecar-schema-DRAFT-v0) and confirm the Docling-per-spine-file architecture (Q-A/Q-B in the in-flight report) — and did he file REVIEWED-55/56? If yes: build the wired extractor (Docling-per-spine-file + inject_epub_footnotes note-layer). If the schema isn't locked: that lock is the gate before 1,284 books write into it — do NOT build the wired extractor against an unlocked schema.

State at wrap: all 8 in-flight tasks complete; chamber-library (CLAUDE.md, runbook, tool-evolution-log modified; build_sidecar.py, verify_body_conservation.py, 2 docs, sidecar-drafts/ new), studium-engine CLAUDE.md, CM-AI register — ALL uncommitted working-tree (steward reviews on return). Dotfiles (PENDING-56 + memory files) committed+pushed at this wrap. Docling installed on M4 (py3.12) + M1.