diff --git a/PENDING.md b/PENDING.md index a06f3ed..b7be68e 100644 --- a/PENDING.md +++ b/PENDING.md @@ -1478,3 +1478,25 @@ Disposition (verbatim capture + executor action-list: `chamber-library/docs/libr **The one flip-condition (for the jurist to test):** if the v1 engine needs *structured critical apparatus (A5) as data* or *DTS-interoperable citation*, revisit toward Option B/TEI. Read: both are later/scholarly needs, not v1 blockers. **Files affected (on execution):** conversion runbook + graduation-spec (Docling front-end; sidecar as graduated structural layer) — a v2.x amendment, change-class PROPOSAL; new `.meta.json` sidecar extractor; `ocrmac_pdf`/EPUB cruft-strippers retired; reprocess track B reshaped to Docling→sidecar-extract→verify→graduate. **Awaiting:** ~~Jurist ratification~~ **JURIST RATIFIED 2026-07-12** — Option A + D1, on independently-verified reasoning (flip-condition tested directly, NOT triggered for v1: voice-manifest = interpretive-persona not manuscript-variant; the corpus's strongest TEI case, Loeb, needs anchors [B2, which A provides] not apparatus [A5]). **Two schema-gating questions CLOSED with substrate evidence (brief §8):** Q1 — Docling emits flat reference-links not native footnotes (0 `[^N]`, 48 `[*](#…)`, `epub_backend` ignores `epub:type`) → the footnote-recognizer arc is REPOSITIONED as the sidecar's note-extraction layer, not retired; Q2 — the Loeb DSL is typed (`[c dimgray]`/`[c darkmagenta]`/`[sup]`, 2.15M+ spans) and the current extractor *flattens* it → **CARDINAL PRINCIPLE set before schema lock: the extractor PRESERVES source typing into typed sidecar fields, never flattens a source-structured distinction to prose** (makes TEI-migration mechanical-by-construction). **REVIEWED-56 can be placed; sidecar schema unblocked to finalize on these two principles.** (Parallel governance still owed: REVIEWED-55 placement.) + +## PENDING-57 — Enforce `body_word_conservation` at graduation (the converter-agnostic verbatim trust gate) +**Date:** 2026-07-12 +**Tag:** [PROPOSAL] +**Summary:** Wire the built `verify_body_conservation.py` into the graduation gate so graduation ENFORCES the verbatim guarantee (currently `body_word_conservation` is listed in `graduation-spec.yaml` line 136 but `verify_graduation.py` does not enforce it — general non-footnote graduation has no automatic verbatim backstop). +**Rationale:** The trust foundation the whole reprocess stands on, and converter-agnostic — it certifies verbatim fidelity whether text came from Docling, pandoc, the Loeb DSL extractor, or OCR. Every tier's reconvert (Loeb B2, the EPUB/PDF Docling reprocess) graduates *through* it; without it "verified" is asserted, not checked. The PENDING-52 follow-on the jurist flagged twice; named in REVIEWED-55 as the gap "now addressed by PENDING-56." +**Method (per-tier, already RATIFIED by the register's §Tiering&Fence):** V-DSL (Loeb) = re-extract deterministically + no-fabrication vs the DSL source; V-TEXT (born-digital) = re-convert + prose-word guard (BUILT — caught jacobs 888 / manguel 2856 / montaigne fn-id residue that audit_cruft missed); V-SCAN = abstain (no ground truth; honest "cannot verify", never a false PASS). Three-tier verdict gates: FLAG blocks · REVIEW holds for curator drop-vs-keep identification · PASS proceeds. +**Options:** (a) enforce all three tiers now; (b) enforce V-TEXT now (built) + wire V-DSL with the Loeb reconvert + V-SCAN abstain. Recommend (b) — lands the built path immediately, wires V-DSL where it's first needed (the Loeb run), no false claims on scans. +**Full brief (for the jurist editor-gate):** `chamber-library/docs/verbatim-gate-enforcement-FOR-JURIST-2026-07-12.md`. +**Change-class:** PROPOSAL — inserts a new blocking check into the graduation gate. Runs the amendment process (executor drafts → jurist editor-gate → steward ratifies → land by supersession into `graduation-spec.yaml` + `verify_graduation.py`). Does NOT change the convention checks, the §V guarantee text, or any tier's evidence-class; V-SCAN gains an honest abstention, not a new claim. +**Files affected (on ratification):** `verify_graduation.py` (add the per-tier body-conservation gate), `graduation-spec.yaml` (make the listed gate enforced + declare the per-tier dispatch), `verify_body_conservation.py` (V-DSL re-extract path — **BUILT**), plus a **retroactive-sweep** producing a remediation queue across the already-graduated corpus (§6 below). +**UPDATE 2026-07-12 (jurist PARTIAL ruling + executor response):** +- **Jurist RATIFIED the architecture** (FLAG/REVIEW/PASS mapping, V-TEXT, V-SCAN abstention, change-class PROPOSAL — distinct from the REVIEWED-55 refusal-only-FIX exception because a FLAG/unresolved-REVIEW has no already-trusted fallback route to canon). **Withheld final ratification pending a V-DSL DEMONSTRATION** (his lesson: simpler-sounding architecture doesn't get trusted a third time on sounding simpler). +- **V-DSL now DEMONSTRATED (§3.5 of the brief) — the demo vindicated the caution:** naive reuse of the V-TEXT k-gram check FALSE-FLAGGED a clean Apollonius (1070 "fabrications", only 6 truly-new = the MD subtitle; the rest = coverage misses from the DSL→MD REFLOW). Corrected → **reflow-tolerant word-multiset** on source-content body: Apollonius (verse) fab=0 PASS · Plato Gorgias (prose) fab=0 PASS · injected fabrication → FLAG with teeth. Fleet fixture added (test_tools 105→109). Honest residual: fabrication half gate-ready; content-vs-boilerplate DROP side delegates to `extract_loeb_dsl`'s integrity report. +- **Provenance verified** (jurist flagged the "already RATIFIES" claim): spec v2.0.0 §Tiering & Fence IS ratified (RATIFIED-SHAPE · OPERATION-BLOCKED) — this PROPOSAL is the un-blocking; the dispatch is confirmed, not originated. +- **§6 RETROACTIVE clause FOLDED IN (steward-directed, rule together):** on ratification the gate runs BACKWARD across graduated canon (V-DSL 952 · V-TEXT 239), diagnostic-first → a remediation queue (FLAG=re-verify→reconvert; REVIEW=curator; PASS=re-attest), NOT silent de-graduation. Motivated by the jurist's 3-for-3 (jacobs/manguel/montaigne all graduated-clean, all failed the new check). Companion efficiency noted (not a condition): the LOCKED sidecar's `placement:back`+`role` resolves the REVIEW-tier endnotes-keep/index-drop ambiguity declaratively for any book carrying a sidecar. +**UPDATE 2026-07-12 (jurist FINAL ruling — RATIFIED on 2 conditions; both now addressed):** +- **Jurist RATIFIED the whole PROPOSAL** — architecture, V-TEXT, V-SCAN, V-DSL's fabrication half (now genuinely demonstrated), and the §6 retroactive sweep (well-precedented by REVIEWED-55's own sweep). Two conditions + one flag: +- **Condition 1 (V-DSL loss half) — ADDRESSED:** the jurist noted loss delegated to `extract_loeb_dsl`'s self-report without the fabrication half's independent injection proof → route loss to REVIEW until a loss-injection test closes the asymmetry. **Built the independent loss teeth + demonstrated:** boilerplate-vocab classifier → clean Apollonius `lost_content=0 PASS`; drop a content word (`banquet`) → `FLAG` naming it. Fleet fixtures both sides (test_tools 105→111). **Honest bound stated:** blind to a dropped *occurrence* of a boilerplate-vocab word (count masked) — the narrow edge the jurist's REVIEW-routing guards; his call whether demonstrated-teeth suffices for auto-PASS or REVIEW-routing is kept for that edge. +- **Condition 2 (43-book census gap) — CONFIRMED:** census = V-DSL 952 · V-TEXT 239 · V-SCAN 50 · V-SUSPECT 29 · V-NONE 14 = 1,284; the 43 = V-SUSPECT+V-NONE = the **fenced-out** tier (no confirmed source) → correctly EXCLUDED from the conservation sweep (can't re-verify a book with no source). Not a gap; the fence working. +- Brief updated (§3.5 loss teeth + §6 census). Change-class PROPOSAL confirmed (distinct from REVIEWED-55's refusal-only-FIX: a FLAG/unresolved-REVIEW has no already-trusted fallback to canon). +**Awaiting:** jurist confirmation the two conditions are closed (+ his call on loss auto-PASS vs REVIEW-routing for the masked-vocab edge) → steward ratification (REVIEWED-57) → land by supersession into `graduation-spec.yaml` + `verify_graduation.py`. diff --git a/REVIEWED.md b/REVIEWED.md index 185a1f9..a5fcf35 100644 --- a/REVIEWED.md +++ b/REVIEWED.md @@ -486,3 +486,72 @@ DONE + DEPLOYED (a9e5196). - **§6 — governance placement: yes** (fleet tool + `test_tools` fixtures + `graduation-spec.yaml` `converted_with` note). - **§7 — mega-EPUB provenance (Jung): OPEN, non-blocking, steward+jurist together.** Jurist leans per-volume (Jung is cited by CW volume; single file fights use) provided the safeguard is airtight (each volume `source`/`source_verified` → the single megavolume sha256; frontmatter states extraction, never poses as a standalone Bollingen edition). Executor fact-check: the corpus does BOTH; **per-volume precedent EXISTS** (Alexander *Nature of Order* 4 files · Lacroux 2 · Habermas/Burney/Camus vol-named). Split is proven content-neutral, so the choice carries no conversion risk — a usability/honesty question. *[Steward + jurist to settle; governs re-converting the legacy Jung `.md`, 0 notes / pre-spec frontmatter.]* **If AUTHORIZED:** BUILD-IN DONE under fleet discipline — `scripts/inject_epub_footnotes.py` (+ `--validate`, refuse-on-mismatch) + `scripts/convert_mega_epub.py`; `test_tools` fixture (fleet 73→77); `graduation-spec.yaml` + `conversion-runbook.yaml` (`policy.footnote_preprocessor`) + `tool-evolution-log.md`. Full brief: `docs/epub-footnote-preprocessor-FOR-JURIST-2026-07-10.md`. Deferred: recognizer coverage of remaining families; per-volume boundary detection (§7); ratified-spec §V cross-ref (runs the `.md` amendment lane, NOT landed here); re-convert legacy Jung/Donne. Tag follow-ons REVIEWED-54. + +## REVIEWED-55 — EPUB footnote pre-processor: end-to-end verify + recognizer arc +**Date:** 2026-07-12 +**Decision:** AUTHORIZED (as ruled by the jurist; steward-confirmed) +**Corresponds to:** PENDING-55 · Jurist ruling `chamber-library/docs/end-to-end-verify-and-nested-block-JURIST-RULING-2026-07-12.md` + +**Notes:** Ratifies the jurist's ruling on the footnote arc: +- **Q1a — end-to-end verify RATIFIED, FIX-class** on the refined test: a §V-chain component is FIX when it (i) only ADDS refusals, (ii) never alters text, (iii) falls back to an already-trusted path. `verify_end_to_end` satisfies all three — turns the guarantee from *verbatim-safe by construction* to *by verification*. +- **Q1b — MANDATORY at graduation** (`--no-verify` forbidden at graduation while `body_word_conservation` is unenforced), PERSISTS after that gate lands (defense-in-depth), **+ RETROACTIVE re-verification required.** [Retroactive sweep DONE (`6c9ca43`): 87 covered books · 42 clean / 45 dirty; **corpus SAFE — 0 dirty graduated**; recognizer's reliable coverage ≈42/87.] +- **Q1c — marker-exclusion made POSITIONAL** (DONE, `e8eeea7`). +- **Q2 — nested-block FIX RETRACTED, honest-refuse CONFIRMED** (the widen pairs but converts dirty; the end-to-end verify caught it; reverted). A fresh PROPOSAL required if ever revisited. Jurist generalization: *structural safety = provisional FIX; end-to-end verbatim proof on real books = final.* +- **Broader flag (jurist):** `body_word_conservation` declared-but-unenforced is the same gap-shape PENDING-52 closed, recurring → now addressed by PENDING-56 (the general verbatim gate = the converter-agnostic trust layer). + +**If AUTHORIZED:** Proceed; tag commits REVIEWED-55. Open follow-ons: wire mandatory-at-graduation (flow must not pass `--no-verify`); the recognizer-maturity decision (improve over-pairing vs accept 42-clean + fallback) is now **subsumed by PENDING-56** — the recognizer repositions as the sidecar note-extraction layer. + +--- + +## REVIEWED-56 — Canonical format: adopt Docling + complete MD-canonical + TEI-mirroring structural sidecar +**Date:** 2026-07-12 +**Decision:** AUTHORIZED +**Corresponds to:** PENDING-56 · Jurist RATIFICATION 2026-07-12 · Brief `chamber-library/docs/canonical-format-decision-FOR-JURIST-2026-07-12.md` + +**Notes:** Steward-decided on the four-tier Docling trial evidence (AskUserQuestion, 2026-07-12); jurist RATIFIED on independently-verified reasoning (not deference): +- **Option A** — canonical = clean Markdown (Docling-converted, verify gates on top) + a `.meta.json` structural sidecar whose vocabulary MIRRORS TEI (`front`/`body`/`back`; `note @place=foot|end`; `milestone`/anchor; reserved `app` slot; `pb`/page-provenance). +- **D1 — adopt Docling** as the EPUB/PDF/scan conversion front-end (MIT, local, per-element provenance); production OCR on CapableHands (M4). Docling **crosses the A3 OCR frontier** `ocrmac` could not. +- **Change-class: PROPOSAL** (changes what graduation produces; touches engine consumer contract §X; whole corpus). +- **Flip-condition tested by the jurist, NOT triggered for v1:** the engine needs neither structured critical-apparatus (A5) as data nor DTS-interoperability in v1 (voice-manifest = interpretive-persona per PENDING-53, not manuscript-variant; the Loeb tier's Stephanus/Bekker need is anchors [B2, which A provides via the sidecar anchor-map], not apparatus). TEI = documented, revisitable future-migration path with those two named triggers. +- **Two schema-gating questions CLOSED before schema lock (brief §8), from substrate evidence:** + - **Q1 — recognizer arc REPOSITIONED, not retired:** Docling emits flat reference-links (0 native `[^N]`, 48 `[*](#…)`; `epub_backend` never reads `epub:type`) → this week's 8-family detection + cross-file anchor resolution + end-to-end verify becomes the sidecar's note-extraction layer; the verify stays as the converter-agnostic verbatim gate. + - **Q2 — CARDINAL EXTRACTOR PRINCIPLE:** the Loeb DSL is typed (`[c dimgray]` 2.15M · `[c darkmagenta]` 471K · `[sup]`) and the current extractor *flattens* it → the extractor MUST **preserve every source-declared typed distinction into a typed sidecar field, never flatten to prose** — makes TEI-migration mechanical-by-construction. + +**If AUTHORIZED:** Proceed on the architecture; tag commits REVIEWED-56. +1. Adopt Docling as the conversion front-end; production OCR on CapableHands. +2. Author the `.meta.json` sidecar schema on the two cardinal principles. +3. Build the sidecar extractor (EPUB metadata + Docling PDF labels + note-linkage), honest-flag undeclared roles; prove on montaigne (EPUB2) + jacobs (EPUB3). +4. Keep the trust gates (`verify_body_conservation`, `normalize_ocr` verbatim-guard) on Docling output. +5. Loeb B2: rewrite `extract_loeb_dsl` → MD + sidecar, preserving the DSL color/`[sup]` typing. +6. Spec: v2.x amendment (Docling front-end + sidecar as graduated structural layer) — change-class PROPOSAL, runs the amendment process; retire `ocrmac_pdf` + EPUB cruft-strippers; reshape reprocess track B. +**Per-content check deferred to the B2 build (not a blocker):** whether `[c darkmagenta]` is apparatus-variants vs bilingual source-text — resolve by sampling volumes; the preserve principle covers both. **[RESOLVED 2026-07-12 by substrate probe: `[c darkmagenta]` = Loeb-reader-RENDERED headers (author/work/book), NOT apparatus; `[c dimgray]` = rendered line-anchors + page-markers + `[sup]` refs. The colors are the reader's rendering, usable as reliable role-CUES, attested honestly as `loeb-render:*`.]** + +--- + +## REVIEWED-56 — LOCK ADDENDUM (whole-schema lock ruling, 2026-07-12) +**Date:** 2026-07-12 +**Decision:** AUTHORIZED — whole-schema lock CONFIRMED (schema-side, unconditional) +**Corresponds to:** PENDING-56 (schema-lock) · Jurist RULING #2 `chamber-library/docs/...B2-evidence-and-whole-schema-lock-JURIST-RULING-2026-07-12.md` · Executor briefs `docs/sidecar-schema-FOR-JURIST-2026-07-12.md` + `docs/sidecar-schema-lock-addendum-B2-evidence-2026-07-12.md` + +**Notes:** Second jurist ruling in the PENDING-56 arc — the schema LOCK (the first ruling ratified the format/architecture; this locks the `.meta.json` sidecar schema itself). Jurist confirmed on independently-checked B2 evidence, not deference: +- **Whole-schema lock CONFIRMED, unconditional on the schema side** → DRAFT (`docs/sidecar-schema-DRAFT-v0-2026-07-12.md`) flips to LOCKED. B2 (`scripts/build_loeb_sidecar.py`) demonstrated flag-and-hold *in code* on the corpus's densest apparatus volume (Ennius FRL Vol I: 44 concordance tables held as typed rows; Diophantus: 38 glyph char-images held with filename preserved) — read back from emitted JSON, not asserted. Every field held; **no missing field**. +- **§3 two minor items CONFIRMED, no schema impact:** (a) `source_declared: "loeb-render:*"` — a new *value* in the existing field's vocabulary, honestly naming a render-cue (weaker than `epub:type`); (b) `app[]` interior shapes `{kind:"table"}` + `{kind:"glyph"}` — principle-5 reserved slot resolved from real evidence. +- **§4 additive-only amendment discipline CONFIRMED + ONE ADDITION:** LOCKED = meaning/shape of every existing field (books trust them); FLEXIBLE (FIX-class, amendment lane) = new optional fields + new `app[]` kinds (backward-compatible; a pre-addition book simply lacks it); PROPOSAL = changing a locked field's meaning **— AND field REMOVAL** (jurist addition: a presence-checking consumer breaks identically from removal as from meaning-change; same silent-retroactive risk → same PROPOSAL treatment). +- **ONE condition MOVES (not closes), extractor-side, does NOT block the lock:** the jurist distinguishes apparatus *density* (what Ennius maximizes) from *diversity of kind*. "Densest volume fit the schema" ≠ "worst case seen" — a sparser volume among 952 could carry a third form (marginal sigla, interlinear glosses; already present in the corpus as `