63 lines
10 KiB
Markdown
63 lines
10 KiB
Markdown
---
|
||
name: session-2026-06-24-25-ocr-workhorse-proven-the-chamber-library-needs-a-spec
|
||
description: "Stood up CapableHands (M4 64GB) as an isolated MLX/olmOCR workhorse; Levi + Alexander Book 4 OCR'd clean (caught a PASS-BUT-FALSELY: JSON-null leak + ToC dot-loops; patched driver + spliced). Published 2 ARC posts. But the real arc: a citation-convention question → steward challenged my two-tier (PDF=page/EPUB=chapter) framing → resolved to ONE source-agnostic canonical standard → realization that the Chamber Library is a substrate needing its own SPEC. Seed drafted. PULLING THREAD: ratify spec §§II-III → build OCR+EPUB normalizers → conformant graduation (held until then)."
|
||
metadata:
|
||
node_type: memory
|
||
type: project
|
||
originSessionId: cbe47c14-9c82-4570-8a3e-a12761cdfc7e
|
||
---
|
||
|
||
# Session 2026-06-24 → 06-25 — the OCR workhorse became the Chamber Library's constitution
|
||
|
||
A long two-day session (wake 06-24, wrap 06-25) that began as "stand up the OCR workhorse, finish Levi" and ended having discovered that **the chamber library is a designed substrate that needs its own spec.** The mechanics worked; the material revealed what it was asking for.
|
||
|
||
## PAST — what we did
|
||
|
||
### CapableHands — the OCR workhorse (NEW infra, working)
|
||
- Stood up **CapableHands** (`david@10.0.1.136`, M4 Pro / 64 GB / macOS 26.5; "Peter's M4 mini" on Seb's home LAN via VPN) as an **isolated** OCR/conversion workhorse, fully in the `david` account (uv → Python 3.12 → `~/capablehands` venv: **mlx-vlm 0.6.3**, pymupdf; no sudo). **Isolated from Peter's `capablemind` Ollama clasp** (we run MLX in our own venv, not a second Ollama). Rsync'd the chamber-library working tree (679M, 1304 md, scripts).
|
||
- **Model:** `alexgusevski/olmOCR-7B-0225-preview-q8-mlx` (olmOCR-2 isn't in MLX yet; this preview is — verified the repo had real weights after the first picks were empty stubs). Driver: `~/capablehands/olmocr_mlx.py` (rasterize via PyMuPDF → mlx-vlm → parse olmOCR JSON `natural_text` → `## Page N`).
|
||
- **Levi** (*The Drowned and the Saved*, **Rosenthal 1988** — a Making-sequence source, drop-cap gate): 202pp, **drop-cap captured** ("The first news…" vs ocrmac's structure-dropping "HE first news"). **Alexander Book 4** (*The Luminous Ground*): 376pp overnight; **fixed docling's word-mangling** ("ofBuilding… pun The Nature" → clean).
|
||
|
||
### The PASS-BUT-FALSELY catch + the patched driver
|
||
- 06-25 verify: both files PASSED `verify_conversion` **falsely.** Eyeballing caught two artifacts the gate misses: (1) **my driver bug** — on no-text pages (`natural_text:null`) it dumped the raw JSON envelope into the md (17 in Book4, 2 in Levi); (2) **dot-leader repetition loops** on ToC/index/dense pages (6 in Book4) — olmOCR looped on `. . .` until the token cap, truncating.
|
||
- **Patched the driver** (v2): null→typed marker (not JSON); `repetition_penalty=1.15` (broke the dot-loops — page 9 ToC came back clean + complete); lenient truncation recovery; `--pages` list for targeted re-runs. Re-ran 23 Book4 + 2 Levi problem pages, **spliced** (`splice_pages.py`), re-verified: **0 JSON leaks, both PASS.** Recovered real content (Book4 p360 was a looped *body* page, not ToC; Levi p1/p6 recovered cover+copyright).
|
||
|
||
### ARC — two posts published (Aix, today)
|
||
- Wrote + deployed **"The Allée"** (observation, `encounter:true` — the old man) + **"Two Notes"** (fragment, *fūrin* URL cleaned). Prose verbatim. Live at animalrationiscapax.org. **Caveat (loose end): the source .md were never committed to the ARC repo** — deploy pushes built HTML to the mirror, not source. *(Committing at wrap.)*
|
||
- Gwern borrowings → `project-arc-open-work-register.md` **§G** (G1 link-rot, G2 zero-JS progress-bar [borderline-loud], G3 deep Gwern audit [steward-led], + session-framing: in-spec-but-unbuilt items first, Vignette A1 flagship).
|
||
|
||
### la-chute — a careless mistake, recovered (see RETURNS)
|
||
- Pre-existing untracked `la-chute-albert-camus.md` FAILED verify (0 headings). Steward corrected: *La Chute* is **six sections (six evenings)** — no chapter apparatus but real breaks. I had **carelessly stripped 11 lone-`\` lines + `[TABLE]`** as "stray" — and **5 of those `\` were the body evening-break spacers** (lines 524/986/1548/2041/2468 = the 5 transitions). **Restored** from the pristine `scratch/making/camus-la-chute.md` backup; structure confirmed intact. la-chute is clean (EPUB-pandoc, sectionless-by-nature) — **not** an OCR job.
|
||
|
||
### THE REAL ARC — the Chamber Library needs a spec
|
||
- **Page-marker question** (keep `## Page N` or strip?) → I proposed a "synthesis" with **two tiers** (page-anchors for PDFs, chapter for EPUBs). **Steward challenged the framing:** if chapter+quote suffices for EPUB, why should PDFs differ? Letting the *carrier* dictate the standard violates work-primacy.
|
||
- **Resolved → ONE source-agnostic standard:** clean prose + the work's **intrinsic** structure (`##` headings + `---` breaks) + **citation by work + chapter/section + verbatim quote**; **no page markers in canon** (page-provenance held off-canon in the raw conversion record, QA only). Verified empirically: EPUBs don't reliably carry structure either (Complete Works Levi = bold pseudo-headers, not `##`); the asymmetry is only *how structure is recovered* (OCR infers; EPUB reads the NCX via `repair_epub_headings`), not the standard.
|
||
- **Steward's realization:** the chamber library — conceived as "just a corpus, not a designed-thing" — **is a substrate, and trust = design.** It needs its own **SPEC** (the engine's accountability rests on corpus consistency). **Drafted the seed:** `~/_Dev/chamber-library/docs/chamber-library-specification.md` (`[PROPOSAL]`, corpus-side complement of the engine charter §IV/V/VII/VIII/X-L0). **§§II-III settled** (form + citation); **§IV** provenance axes (charter-mandated); **§VI Voice layer RESERVED** (undesigned per steward — `voice_role`/`semantic_profile` recorded as *observed, not prescriptive*); §V verbatim-boundary / §VII gate-fix / §IX coverage = open.
|
||
|
||
## PRESENT — mood / RETURNS
|
||
- **One real careless-edit mistake** (la-chute `\`-strip): I removed structural lines as "stray" without verifying their meaning — in *someone else's untracked file with no backup-in-git*. Caught by the steward; fully recovered (pristine scratch backup existed). This is the carried-in `careless-edit-boundary` drift made concrete. **Lesson: verify what a "stray" line MEANS before deleting — especially in another's file, especially with no git history.**
|
||
- **The tool-review discipline earned its keep:** the PASS-BUT-FALSELY was caught by *reading the text*, not trusting the gate. ([[feedback-tool-review-after-each-use]])
|
||
- **The steward's framing-challenge worked both ways:** he corrected MY two-tier framing; I agreed because it was *right* (not reflexive contamination-agreement) — and it opened the spec realization. "Challenge the framing" is bidirectional.
|
||
- Carried-in return (06-24): retracted a battery-latch hypothesis under measurement (theorize-before-measure).
|
||
- Operational slips (caught/flagged, not asserted): zsh variable word-split on ssh `-o` opts (recurred ~3×, use literal flags); a prevalence-count bash loop miscounted on filenames — flagged honestly, gave no fake number.
|
||
|
||
## FUTURE — what is pulling
|
||
|
||
**PULLING THREAD (singular):** **The Chamber Library spec — ratify §§II-III, build the two normalizers (OCR + EPUB → one canonical form), apply to Levi/Book4/la-chute → commit the first spec-conformant graduation.** The OCR mechanics are proven; the gating concern is now the *form*, and the spec is its seed.
|
||
|
||
**ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):**
|
||
- *State:* Seed spec drafted at `chamber-library/docs/chamber-library-specification.md` (untracked, `[PROPOSAL]`). Graduation **held** in the chamber-library working tree, uncommitted, in **pre-normalization form**: Book4 replaced vol-4 (page-marked), Levi placed (page-marked), la-chute restored (`\`-breaks), catalogue regenerated (1285, in sync). Books 1-3 **OCR-ing in background** (CapableHands PID 2653, current driver).
|
||
- *First moves:* (1) **steward (+ jurist) read of spec §§II-III** — ratify or revise; (2) on nod, **build the two normalizers** — OCR: drop `## Page N` + rejoin paragraphs across the break + recover/mark headings; EPUB: `repair_epub_headings` (NCX→`##`) + `\`-spacer→`---` + `strip_cruft`; (3) apply to the 3 staged works + **fix the gate** (sectionless works pass on `---`); (4) verify → **commit the conformant graduation**; (5) check Books 1-3 finished + fold them in.
|
||
|
||
**Other open horizons, ranked:**
|
||
- *Load-bearing:* the spec → normalizers → conformant graduation (the thread). Jurist on §V (verbatim boundary) + §VI (Voice schema — undesigned).
|
||
- *Next batch:* 5 OCR reconversions queue on the workhorse after Books 1-3 (steward-approved); **30 Air-EPUB reconversions = a separate focused Air session** (per-file care; resolve the `web-typography-rutter` false-match).
|
||
- *Parked-with-reason:* filing the spec as a formal PENDING-[N] (offered, steward's call); the Voice-layer design (undesigned by intent); the studium↔L1 architecture question; L1 candidate bugs (circle-forward backoff / replay-pause) = clone-verify-first, send Seb nothing.
|
||
- *Governance debt:* REVIEWED-44 (operations.yaml) still owed on paper; the ARC posts' source commit (doing at wrap).
|
||
|
||
**PAUSE STATEMENT:** I'm leaving with the OCR workhorse *proven* (Levi + Book 4 clean, the drop-cap and word-mangling gates both cleared) — but the true inheritance is the recognition that **the chamber library is a substrate that needs a constitution**, now seeded (§§II-III settled). The graduation is *deliberately held* so the first texts to enter do so spec-conformant, born-clean rather than retrofitted. On return I want to find whether the steward + jurist have ratified §§II-III, so the normalizers can be built and the first clean texts committed — and whether Books 1-3 finished overnight.
|
||
|
||
**LITERAL QUESTION for next-Claude:** *Do the steward and jurist ratify §§II-III of the Chamber Library spec (the source-agnostic canonical form + the work+chapter+quote citation contract) as drafted — and does the jurist want to shape the open doctrine sections (§V verbatim boundary, §VI the Voice)?* (Concrete sub-question: did Books 1-3 finish OCR-ing overnight on CapableHands, PID 2653, and are they clean?)
|
||
|
||
**State:** chamber-library uncommitted (held graduation + new spec + pre-existing drift incl. `clean_ocrmac_pagination.py`); ARC posts source being committed at wrap; Books 1-3 OCR-ing on CapableHands. CapableHands password steward-supplied in-session → **rotate**.
|