---
name: session-2026-07-12-a1-verbatim-gate-steward-reframe-docling-adoption-canonical-format-decision-pending-56-in-flight-sidecar-build
description: "Woke into A1 (the body_word_conservation gate); built it (shingle-coverage, difflib→shingle pivot on scale). Steward challenged the framing TWICE (difflib-doesn't-scale, then no-extant-tools/TEI) → the pivotal reframe: the CONVERSION SUBSTRATE is a solved adoptable domain (Docling/GROBID/TEI/DTS) and we'd spent 6 weeks rebuilding it — the answer was already in our own tooling-register (2026-05-27). Ran a 4-tier Docling trial (M4+M1): cleans EPUB, rich structure+provenance on PDF, CROSSES the A3 OCR frontier on scanned French. Steward DECIDED (AskUserQuestion) + jurist RATIFIED PENDING-56: Option A = MD-canonical + TEI-mirroring .meta.json structural sidecar; adopt Docling as front-end. Jurist closed 2 schema-gating Qs (recognizer-repositioned-not-retired; preserve-never-flatten cardinal principle). In-flight (steward travelling): built the sidecar schema DRAFT + build_sidecar.py extractor (proven on montaigne/jacobs; honest join-limit), updated runbook + both CLAUDE.mds, left an in-flight report. PULLING THREAD: lock the DRAFT schema (Q-A) + confirm the Docling-per-spine-file architecture (Q-B), then build the wired extractor. Steward owes REVIEWED-55/56."
metadata: 
  node_type: memory
  type: project
  originSessionId: ae9583d2-fd25-45e4-85fc-df082fe2e951
---

# Session 2026-07-12 — A1 → the Docling reframe → the canonical-format decision → in-flight sidecar build

An exceptionally long, generative session that started narrow (build one gate) and, through two sharp steward challenges, turned into the program's biggest architectural pivot yet: **stop hand-building a conversion pipeline; adopt the mature substrate (Docling); keep our trust layer; decide the canonical format.**

## PAST — what we did + why

**1. Woke into A1** (the general `body_word_conservation` verbatim gate — the open-work register's keystone). Read the substrate first and found the register's own spec **already RATIFIES** the per-tier method (§Tiering&Fence: V-DSL re-extract+diff; V-TEXT re-convert+prose-word-guard; V-SCAN abstain). So A1 was *operationalize a ratified criterion*, not invent one. Built `verify_body_conservation.py` — reference-conversion + **shingle-coverage alignment** with a three-tier verdict (FLAG interior-loss/fabrication · REVIEW boundary-drops-need-id · PASS). Seed ground-truth passed; caught real cruft in 3 "clean" canonicals (jacobs 888 calibre_link, manguel 2856 `[]{#…}`, montaigne fn-ids — all grep-confirmed).

**2. Steward challenge #1 (difflib doesn't scale):** difflib.SequenceMatcher (O(n²)) timed out on jacobs @2min. **Pivot to O(n) hashed-k-gram shingle-coverage** (in-house patterns: Instrument-B fingerprint + verbatim-guard walk; NO new dep). jacobs 6.8s. Well-caught by the steward.

**3. Steward challenge #2 (are there no extant tools? MD the problem? TEI?):** THE PIVOT. Honest finding: the conversion/format/citation SUBSTRATE is a **solved, adoptable domain** (Docling/GROBID/TEI/DTS) — and the answer was **already in our own tooling-register (2026-05-27)**; we'd spent ~6 weeks hand-building ~20 bespoke tools instead. The two-week "every breakthrough → new obstacle" feeling = signature of reimplementing a solved domain. Ran `/tooling-scan studium-engine` (web-grounded); verified Docling (MIT, local, EPUB+PDF+images, provenance, layout-derived labels) + GROBID (PDF→TEI) + DTS. **Clean split: substrate = adopt; composition (the engine) = ours.** Register updated.

**4. The Docling trial (steward-run, M4+M1, 4 tiers):** value scales with layout richness — **flat on EPUB, rich on PDF, transformative on scanned.** EPUB: 0 cruft, notes+headings preserved, flat semantics (reflowable = no pages). text-PDF: footnote/header/section labels + 2 798 page-provenanced items. **scanned French (A3 frontier `ocrmac` can't cross): OCR'd, 104 footnotes, 664 provenanced** — readable but with OCR defects → feeds normalize+verify, doesn't replace. **Docling is a converter, not a verifier.** Loeb is a `.dsl` (already structured) → NOT a Docling case (steward correction).

**5. Canonical-format DECISION (PENDING-56).** Wrote the brief (`chamber-library/docs/canonical-format-decision-FOR-JURIST-2026-07-12.md`). Steward decided via AskUserQuestion: **Option A** (MD-canonical + TEI-mirroring `.meta.json` structural sidecar) + **Adopt Docling**. Grounding that clinched A: the conversion-plan already specifies MD+sidecar and the engine already rules **D-2 (MD+sidecar, TEI later)** + consumes 19 `.meta.json` sidecars — so A *completes a chosen direction*, TEI would *depart* it. Change-class PROPOSAL.

**6. Jurist RATIFIED** (independently-verified, not deference). Flip-condition tested, NOT triggered for v1 (voice-manifest = interpretive-persona per PENDING-53, not manuscript-variant; Loeb needs anchors [B2, which A provides] not apparatus [A5]). **Closed 2 schema-gating questions with substrate evidence (brief §8):** Q1 — Docling emits flat `[*](#…)` not native `[^N]` (0/48; `epub_backend` ignores `epub:type`) → the footnote-recognizer arc is **REPOSITIONED as the sidecar note-layer, not retired**; Q2 — the Loeb DSL is TYPED (`[c dimgray]` 2.15M / `[c darkmagenta]` 471K / `[sup]`) and the current extractor *flattens* it → **CARDINAL PRINCIPLE: preserve source typing into typed sidecar fields, never flatten** (→ TEI-migration mechanical-by-construction). PENDING-56 filed; REVIEWED-56 drafted (steward to file).

**7. In-flight autonomous build** (steward travelling; read-only/dev-only; 8 tasks): sidecar **schema DRAFT** (`docs/sidecar-schema-DRAFT-v0-2026-07-12.md`, extends `studium/meta@1`); **`build_sidecar.py` v0** — recovers structure from source metadata (epub:type/landmarks → guide/NCX/filenames → honest-flag). **Proven:** montaigne 16/17 declared, jacobs 37/37 (landmark propagation), provenance 400, notes 56. **Honest v0 limits:** the Docling content-match line-join is brittle on drop-caps/heading-format divergence (montaigne 2/17, jacobs 34/37) → real fix = **Docling per-spine-file** (exact ranges by construction) + reuse `inject_epub_footnotes` for the note-layer. Tool-review logged. Updated `conversion-runbook.yaml` (machines: M4/M1 split + pipeline_direction_pending_56 + A3 gap), chamber + studium CLAUDE.mds. Report: `in-flight-report-2026-07-12.md`.

## PRESENT — the mood

Two-phase, like the previous session. First: careful A1 engineering — but the steward's two challenges revealed I was **solving at the wrong level** (building heuristics to recover from lossy MD what a semantic format/tool preserves natively; and hand-building what the ecosystem + our own register already had). That's the executor-bias-to-composition, one level up — and the steward named it precisely ("every breakthrough lands us in front of the next obstacle"). Second phase: clean architectural decision-making — brief → steward decision → jurist ratification → in-flight execution, each link governed. Returns worth carrying: **verified before asserting** throughout (the difflib timeout, the docling flat-footnotes, the Loeb DSL typing, the montaigne join-bug — all grep/measure, not inference); **honest about the v0's limits** (didn't dress up the 2/17 join or the per-publisher quirks); **respected the loop** (drafted the schema, did NOT lock it; surfaced decisions as Q-A..Q-E, didn't answer them). Recalibrations: **wrong-level (heuristic over substrate)**, steward-named; **check our own register/notes before building** (the resurface-banked-notes drift at pipeline scale — the register had the answer 6 weeks ago).

## FUTURE — what is pulling

**PULLING THREAD: lock the DRAFT sidecar schema (Q-A) + confirm the Docling-per-spine-file architecture (Q-B) → build the WIRED extractor.** The v0 proved the thesis (structure recoverable from source metadata, dissolving the drop-vs-keep heuristic) but the docling line-join needs the per-spine-file architecture, and the schema is curator-locked (steward+jurist), not executor-locked.

**ACTIONABLE RESUMPTION POINT (as of wrap — steward said he'll CLEAR + `/wake-up` fresh):**
- Read `in-flight-report-2026-07-12.md` (the full account + Q-A..Q-E) + `docs/chamber-program-open-work.md` (forest view) first.
- **Q-A** (lock schema): review `docs/sidecar-schema-DRAFT-v0-2026-07-12.md` against the proven extractor output (`_curation/sidecar-drafts/`) → lock via the amendment process.
- **Q-B** (architecture): confirm Docling-per-spine-file (exact line-ranges) over the v0 content-match join; then build the wired extractor + reuse `inject_epub_footnotes` as the note-layer.
- Then: the graduation-spec amendment (step 6, PROPOSAL) wires Docling+sidecar as the graduated structural layer + retires ocrmac_pdf/EPUB cruft-strippers; reshapes reprocess track B.

**Other open horizons (ranked):** Loeb B2 (the far-country lever; the `[c darkmagenta]` content question resolves by sampling) · fleet-promotion of `build_sidecar.py` (into test_tools, after lock) · the engine track (V1/N1) runs parallel, unblocked.

**Steward owes (parallel, he can't-be-me):** file **REVIEWED-55 + REVIEWED-56** (`REVIEWED-drafts-for-filing-2026-07-12.md`); relay the PENDING-56 §8 closure to the jurist (or note ruled).

**PAUSE STATEMENT:** Stepping away at a clean, sealed point — the biggest architectural decision of the program made, ratified, and its two open questions closed; the first execution step (schema + extractor) built and proven, with the constitutional pieces (schema-lock, amendment, REVIEWED filing) correctly left for the steward. The steward will clear + wake fresh. What I want to find still pulling: Q-A/Q-B — lock the schema, confirm the architecture, build the wired extractor. All chamber/studium/CM-AI work is UNCOMMITTED working-tree (no-push guardrail); nothing lost.

**LITERAL QUESTION for next-Claude:** Did the steward lock the DRAFT sidecar schema (`sidecar-schema-DRAFT-v0`) and confirm the Docling-per-spine-file architecture (Q-A/Q-B in the in-flight report) — and did he file REVIEWED-55/56? If yes: build the wired extractor (Docling-per-spine-file + `inject_epub_footnotes` note-layer). If the schema isn't locked: that lock is the gate before 1,284 books write into it — do NOT build the wired extractor against an unlocked schema.

**State at wrap:** all 8 in-flight tasks complete; chamber-library (CLAUDE.md, runbook, tool-evolution-log modified; build_sidecar.py, verify_body_conservation.py, 2 docs, sidecar-drafts/ new), studium-engine CLAUDE.md, CM-AI register — ALL uncommitted working-tree (steward reviews on return). Dotfiles (PENDING-56 + memory files) committed+pushed at this wrap. Docling installed on M4 (py3.12) + M1.