68 lines
11 KiB
Markdown
68 lines
11 KiB
Markdown
---
|
||
name: session-2026-06-29-corpus-graduations-ocr-docling-pipelines-the-clasp-pivot
|
||
description: "Long session, two arcs. CHAMBER: graduated 4 Making works (Taylor/Virilio/Handke/Illich), folded RAW_HTML into audit_cruft (honest gate), launched overnight pipelines (CapableHands olmOCR ×3 + local docling ×3 — Technics&Civ resolved clean), mapped the whole remaining-work landscape (Kindle-EPUB sub-batch, the 72→70 backlog = mostly the reconvert-flagged set, source-matched 24/34 to the library, scoped the metadata indexer + found the 1263-missing-sidecar gap). Then PIVOT to L1: Seb meeting tomorrow → scoped standing up CapableHands as a BMF clasp (43M). PULLING THREAD: the clasp deploy on CapableHands tomorrow MORNING (before the 15:00 Seb call) — priority 1; indexer priority 2."
|
||
metadata:
|
||
node_type: memory
|
||
type: project
|
||
originSessionId: 604b3d1d-6111-48f2-9683-2dfba02d5b6b
|
||
---
|
||
|
||
# Session 2026-06-29 — graduations, pipelines, and the clasp pivot
|
||
|
||
A very long, two-arc day. Most of it was chamber corpus work (continuing the graduation batch from 06-28); the last third pivoted to L1/CapableMind because the steward has a Seb meeting tomorrow at 15:00.
|
||
|
||
## PAST — what we did + decided
|
||
|
||
### Chamber graduations (4, all pushed)
|
||
Continued the Making graduation batch with the proven pandoc→clean→structure→verify→graduate→sidecar→catalogue→commit→push pattern:
|
||
- **Taylor — Ethics of Authenticity** → `critical_modernity/hermeneutics` (NEW node, steward-chosen). The pattern-setter: exposed that footnote conversion was fragmented per-publisher. Added **footnote family B** (`#…_fn-N`/`#…_rfn-N`) to clean_epub_residue + a **loud-abort guard** in clean_pandoc_html_residue (silent-loss class closed) + generalized IMG_LINE by extension (caught a 2nd instance of the same pin-bug). 59 endnotes kept.
|
||
- **Virilio — The Information Bomb** → `critical_modernity/french_theory`. **Footnote family C** (caret-wrapped per-chapter `^([N](#…_chXfnY))`). 72 per-chapter notes.
|
||
- **Handke — A Sorrow Beyond Dreams** → `literature/contemporary`. Continuous memoir, no notes. GUARD LESSON: the prose word-guard validates only the KEPT region, not that the RIGHT region was kept (near-miss: a back-matter phrase also lived in the ToC → dropped the whole memoir, guard still said CLEAN). Added a **total-prose size-sanity check** to structure scripts.
|
||
- **Illich — Deschooling Society** → `contemporary_voices/environmental` (with the existing Illich). No-notes edition; back-matter duplicate-ToC dropped.
|
||
Per-book structure scripts archived in `scripts/archive/conversions/`.
|
||
|
||
### The RAW_HTML fold-in (the owed follow-on, commit 9e99e07)
|
||
Folded the verifier's comprehensive RAW_HTML into `audit_cruft` as the single source; verify imports it (no double-count). The two gates now AGREE. Monotonic. Canonical true-state became HONEST: **1215 clean / 72 with latent cruft** (later 1220/70 after a small clean). Then built `clear_safe_html_residue.py` and cleared the mechanically-safe slice (svg/nav scaffolding) from 11 files — the prose-guard REFUSED 9 (Arden Shakespeare: an untight `<nav>` would've deleted 10,497 words — caught).
|
||
|
||
### Overnight pipelines LAUNCHED (the steward's "they need to run anyway")
|
||
- **CapableHands** (M4 64GB, david@10.0.1.136): `making-ocr-queue.sh` (nohup), olmOCR-7B MLX @200dpi, 3 single-column scans SEQUENTIAL: Jonas (280pp) → Mumford *Myth of the Machine* I (388pp) → II (540pp). Mid-flight at wrap (Jonas ~147/280). Triaged all 8 PDFs first (text-layer→docling, scans→OCR, Levi=done, Psalms_2=flagged).
|
||
- **Local docling** — DONE (all 3 text-layer PDFs in `scratch/making/docling/`): **Technics & Civilization (tlie:0 modem:0 CLEAN — resolves the parked corrupt EPUB!)**, Speed & Politics, Burnout Society. Need light post-processing (`&`, letter-spaced headings, trim, verify, graduate).
|
||
|
||
### The landscape, fully mapped (the valuable triage)
|
||
- **Remaining Making EPUBs (Crawford/Steiner/Han×2/Winnicott/Borgmann/Calasso)** = NOT quick clears. All Kindle-derived (filepos), structure-LOST (validated: ebook-convert+repair = 0/15 headings), varied caret-footnote anchors. A coherent "Kindle-derived, structure-lost" sub-batch needing a structure-recovery TOOL built once, not per-book hand-work.
|
||
- **Alter Psalms PARKED** (character-as-image hazard — Hebrew transliteration + script as inline images; verified by viewing glyphs); **Mumford T&C-EPUB superseded** by the clean PDF; new standing lesson [[feedback-character-as-image-hazard]].
|
||
- **The 72→70 backlog**: cross-referenced against `_curation/reconvert-list-2026-06-16.txt` → **34 are the reconvert-FLAGGED set** (already judged "reconvert, don't patch"), not a cleaning job. Source-matched them against the steward's master library (`~/Documents/___The Library [ePub_AWZ3]/`, 2190 ebooks): **24 found, 10 to locate** (French/edition-named). → `_curation/reconvert-queue-2026-06-29.md`.
|
||
- **Steward's bigger goal**: link the two libraries (provenance) + dedupe the messy folder. → scoped the **metadata indexer** (`_curation/library-indexer-scope-2026-06-29.md`) — one read-only internal-metadata tool serving locate+link+dedupe; edition-identity load-bearing; dedup proposes-never-deletes. [[project-source-library-link-and-dedupe]]
|
||
- **Sidecar gap** (steward flagged): 1282 canonical · 19 sidecars → ~1263 MISSING; ~1073 also lack frontmatter — but VERIFIED the 952 Loebs aren't a void (consistent `# AUTHOR, Work` headers, NEVER YAML, NEVER stripped — DSL pipeline predates the convention; backfillable). It's ONE metadata-completion program: frontmatter-backfill → sidecar-gen → link → dedupe, indexer = keystone.
|
||
|
||
### The L1 pivot (Seb meeting tomorrow 15:00)
|
||
- **CapableMind status**: nothing new from Seb since 06-13. The real change is the steward's own UNCOMMITTED 06-22/06-23 L1 work: **N6 deploy (#175) landed live and HOLDS** — the multi-day wedge gone (CPU 100%→~21%), A1″ migration finally live (vector_chunks 19→26 cols). That's the lead good news for Seb. L1 tracker is STALE (current-state dated 06-06).
|
||
- **The clasp insight** (steward): we never set up the circle/clasp on CapableHands. Verified 43M tiered-circle-inference is BUILT in BMF (CircleInferenceProvider + clasp-* modules), but CapableHands is GREENFIELD (no Ollama/BMF/models). Scoped it: `CapableMind-AI/docs/thinking/David/l1-reliability/clasp-on-capablehands-scope-2026-06-29.md`. This is the design-aligned answer to PENDING-41 (Air saturation).
|
||
|
||
## PRESENT — mood / returns
|
||
- **The recurring shape, again**: each "easy" bucket revealed hard sub-cases on inspection (Making EPUBs → Kindle-structure-lost; the 72 backlog → reconvert-flagged, not cleanable; Bachelard's "sup" → OCR mess). NAMED it honestly to the steward 3×. The pre-check discipline (image-inspect, OCR-sample, prose-guard, reconvert-list-check) earned its keep — caught Alter (would've mutilated Hebrew), Mumford (OCR scan), the Arden 10k-word near-deletion, and that I was about to clean files marked for reconversion.
|
||
- **Returns (load-bearing)**: measure-don't-theorize / verify-against-the-record paid repeatedly. The steward's memory caught me cleaning reconvert-flagged files; their "or was it stripped?" question was the right verification trigger (answer: never-YAML, no data loss).
|
||
- **Honest restraint**: did NOT rush the per-file backlog or the Kindle EPUBs at extreme context length — surfaced "this is dwelling-work, not a sweep" rather than ship rushed graduations. Prime directive enacted.
|
||
- **Recalibration**: I over-projected "easy and doable" three times before triaging properly. Lesson: triage by ACTUAL content before promising a bucket is easy.
|
||
|
||
## FUTURE — what is pulling
|
||
|
||
**PULLING THREAD (singular):** **Stand up CapableHands as a BMF clasp tomorrow MORNING** (Seb is 6h behind, asleep; steward's 15:00 call is the deadline). Pair the M1 Air as a light circle member, validate the 43M offload path → solves PENDING-41 → demo/discuss with Seb. PRIORITY 1. The library indexer is PRIORITY 2.
|
||
|
||
**ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):**
|
||
- *First:* confirm the overnight OCR finished and CapableHands is FREE: `ssh david@10.0.1.136 'tail ~/capablehands/scratch/making-ocr-queue.log; ls -la ~/capablehands/scratch/*.md'`. (3 OCR outputs — Jonas + Mumford ×2 — will be waiting for later chamber post-processing; don't process them now, just confirm done.)
|
||
- *Then:* execute the clasp scope `CapableMind-AI/…/clasp-on-capablehands-scope-2026-06-29.md` — Phase 0 prereqs (verify qwen3.5 is a real Ollama tag or pick fallback WITH-Seb-awareness; install Ollama on CapableHands; clone+build BMF; pull frontier model ~17-20GB) → Phase 1 clasp config (resolve how claspDuties get assigned: read bootstrap.ts:1132 + src/core/pairing/) → Phase 2 Air as member → Phase 3 form circle (the genuine unknown: manifest-manager.ts pairing) → Phase 4 validate offload (bm_circle_inference_* metrics; Air doesn't saturate).
|
||
- *Governance:* deploy from a checkout, NOT BMF main; it's a co-authored experiment validating Seb's design; surface to him at 15:00.
|
||
|
||
**Other open horizons, ranked:**
|
||
- *Load-bearing (chamber, background-complete):* post-process the 3 docling outputs (Technics&Civ/Speed&Politics/Burnout) + the 3 OCR outputs (Jonas/Mumford×2) → graduate. Quality-check Jonas FIRST (tool-review discipline).
|
||
- *Load-bearing (program):* the metadata indexer (priority 2) → unblocks frontmatter-backfill → sidecar-generation (1263 missing) → source-link → dedupe.
|
||
- *Deferred-with-reason:* the Kindle-EPUB sub-batch (needs the structure-recovery tool built first); Alter Psalms (focused session, steward's Hebrew); the 10 to-locate reconvert sources (the indexer auto-finds them); the poetry fenced_div backlog (judgment needed).
|
||
- *L1 housekeeping:* commit the 3 CM-AI L1 docs; refresh the stale L1 tracker (current-state still 06-06, predates the N6 win).
|
||
|
||
**PAUSE STATEMENT:** I am about to be away from this. By morning the OCR will likely have finished (CapableHands free) and nothing else will have moved unless the steward or Seb acted. What I most want to find still pulling: **the clasp ready to stand up on CapableHands** — the first real test of Seb's 43M design on the steward's own hardware, the thing that turns the Air's saturation from a problem into a solved demo for the 15:00 call.
|
||
|
||
**LITERAL QUESTION for next-Claude:** *Can the clasp actually be stood up end-to-end this morning, or do the two genuine unknowns (how claspDuties get assigned, and the circle-pairing procedure in manifest-manager.ts) force it to become the 15:00 conversation-with-Seb instead?* (Sub-q: is `qwen3.5` a real pullable Ollama tag, or do we need a fallback family — and did the OCR finish cleanly overnight?)
|
||
|
||
**State:** chamber `main @ 15ea750` + studium `main @ 844b207`, both pushed, clean. 4 graduations + RAW_HTML fold-in + safe-html-clear all pushed. 3 docling outputs done (scratch), 3 OCR mid-flight on CapableHands. Indexer + clasp both scoped. 3 CM-AI L1 docs uncommitted (for tomorrow). New memories: [[project-source-library-link-and-dedupe]], [[feedback-character-as-image-hazard]], [[project-sidecar-typology-protocol-dependent]].
|