dotfiles/claude/memory/session-2026-06-29-corpus-graduations-pipelines-clasp-scope.md

name, description, metadata

name	description	metadata
session-2026-06-29-corpus-graduations-ocr-docling-pipelines-the-clasp-pivot	Long session, two arcs. CHAMBER: graduated 4 Making works (Taylor/Virilio/Handke/Illich), folded RAW_HTML into audit_cruft (honest gate), launched overnight pipelines (CapableHands olmOCR ×3 + local docling ×3 — Technics&Civ resolved clean), mapped the whole remaining-work landscape (Kindle-EPUB sub-batch, the 72→70 backlog = mostly the reconvert-flagged set, source-matched 24/34 to the library, scoped the metadata indexer + found the 1263-missing-sidecar gap). Then PIVOT to L1: Seb meeting tomorrow → scoped standing up CapableHands as a BMF clasp (43M). PULLING THREAD: the clasp deploy on CapableHands tomorrow MORNING (before the 15:00 Seb call) — priority 1; indexer priority 2.	node_type type originSessionId memory project 604b3d1d-6111-48f2-9683-2dfba02d5b6b

Session 2026-06-29 — graduations, pipelines, and the clasp pivot

A very long, two-arc day. Most of it was chamber corpus work (continuing the graduation batch from 06-28); the last third pivoted to L1/CapableMind because the steward has a Seb meeting tomorrow at 15:00.

PAST — what we did + decided

Chamber graduations (4, all pushed)

Continued the Making graduation batch with the proven pandoc→clean→structure→verify→graduate→sidecar→catalogue→commit→push pattern:

• Taylor — Ethics of Authenticity → critical_modernity/hermeneutics (NEW node, steward-chosen). The pattern-setter: exposed that footnote conversion was fragmented per-publisher. Added footnote family B (#…_fn-N/#…_rfn-N) to clean_epub_residue + a loud-abort guard in clean_pandoc_html_residue (silent-loss class closed) + generalized IMG_LINE by extension (caught a 2nd instance of the same pin-bug). 59 endnotes kept.
• Virilio — The Information Bomb → critical_modernity/french_theory. Footnote family C (caret-wrapped per-chapter ^([N](#…_chXfnY))). 72 per-chapter notes.
• Handke — A Sorrow Beyond Dreams → literature/contemporary. Continuous memoir, no notes. GUARD LESSON: the prose word-guard validates only the KEPT region, not that the RIGHT region was kept (near-miss: a back-matter phrase also lived in the ToC → dropped the whole memoir, guard still said CLEAN). Added a total-prose size-sanity check to structure scripts.
• Illich — Deschooling Society → contemporary_voices/environmental (with the existing Illich). No-notes edition; back-matter duplicate-ToC dropped. Per-book structure scripts archived in scripts/archive/conversions/.

The RAW_HTML fold-in (the owed follow-on, commit 9e99e07)

Folded the verifier's comprehensive RAW_HTML into audit_cruft as the single source; verify imports it (no double-count). The two gates now AGREE. Monotonic. Canonical true-state became HONEST: 1215 clean / 72 with latent cruft (later 1220/70 after a small clean). Then built clear_safe_html_residue.py and cleared the mechanically-safe slice (svg/nav scaffolding) from 11 files — the prose-guard REFUSED 9 (Arden Shakespeare: an untight <nav> would've deleted 10,497 words — caught).

Overnight pipelines LAUNCHED (the steward's "they need to run anyway")

• CapableHands (M4 64GB, david@10.0.1.136): making-ocr-queue.sh (nohup), olmOCR-7B MLX @200dpi, 3 single-column scans SEQUENTIAL: Jonas (280pp) → Mumford Myth of the Machine I (388pp) → II (540pp). Mid-flight at wrap (Jonas ~147/280). Triaged all 8 PDFs first (text-layer→docling, scans→OCR, Levi=done, Psalms_2=flagged).
• Local docling — DONE (all 3 text-layer PDFs in scratch/making/docling/): Technics & Civilization (tlie:0 modem:0 CLEAN — resolves the parked corrupt EPUB!), Speed & Politics, Burnout Society. Need light post-processing (&, letter-spaced headings, trim, verify, graduate).

The landscape, fully mapped (the valuable triage)

• Remaining Making EPUBs (Crawford/Steiner/Han×2/Winnicott/Borgmann/Calasso) = NOT quick clears. All Kindle-derived (filepos), structure-LOST (validated: ebook-convert+repair = 0/15 headings), varied caret-footnote anchors. A coherent "Kindle-derived, structure-lost" sub-batch needing a structure-recovery TOOL built once, not per-book hand-work.
• Alter Psalms PARKED (character-as-image hazard — Hebrew transliteration + script as inline images; verified by viewing glyphs); Mumford T&C-EPUB superseded by the clean PDF; new standing lesson feedback-character-as-image-hazard.
• The 72→70 backlog: cross-referenced against _curation/reconvert-list-2026-06-16.txt → 34 are the reconvert-FLAGGED set (already judged "reconvert, don't patch"), not a cleaning job. Source-matched them against the steward's master library (~/Documents/___The Library [ePub_AWZ3]/, 2190 ebooks): 24 found, 10 to locate (French/edition-named). → _curation/reconvert-queue-2026-06-29.md.
• Steward's bigger goal: link the two libraries (provenance) + dedupe the messy folder. → scoped the metadata indexer (_curation/library-indexer-scope-2026-06-29.md) — one read-only internal-metadata tool serving locate+link+dedupe; edition-identity load-bearing; dedup proposes-never-deletes. project-source-library-link-and-dedupe
• Sidecar gap (steward flagged): 1282 canonical · 19 sidecars → ~1263 MISSING; ~1073 also lack frontmatter — but VERIFIED the 952 Loebs aren't a void (consistent # AUTHOR, Work headers, NEVER YAML, NEVER stripped — DSL pipeline predates the convention; backfillable). It's ONE metadata-completion program: frontmatter-backfill → sidecar-gen → link → dedupe, indexer = keystone.

The L1 pivot (Seb meeting tomorrow 15:00)

• CapableMind status: nothing new from Seb since 06-13. The real change is the steward's own UNCOMMITTED 06-22/06-23 L1 work: N6 deploy (#175) landed live and HOLDS — the multi-day wedge gone (CPU 100%→~21%), A1″ migration finally live (vector_chunks 19→26 cols). That's the lead good news for Seb. L1 tracker is STALE (current-state dated 06-06).
• The clasp insight (steward): we never set up the circle/clasp on CapableHands. Verified 43M tiered-circle-inference is BUILT in BMF (CircleInferenceProvider + clasp-* modules), but CapableHands is GREENFIELD (no Ollama/BMF/models). Scoped it: CapableMind-AI/docs/thinking/David/l1-reliability/clasp-on-capablehands-scope-2026-06-29.md. This is the design-aligned answer to PENDING-41 (Air saturation).

PRESENT — mood / returns

• The recurring shape, again: each "easy" bucket revealed hard sub-cases on inspection (Making EPUBs → Kindle-structure-lost; the 72 backlog → reconvert-flagged, not cleanable; Bachelard's "sup" → OCR mess). NAMED it honestly to the steward 3×. The pre-check discipline (image-inspect, OCR-sample, prose-guard, reconvert-list-check) earned its keep — caught Alter (would've mutilated Hebrew), Mumford (OCR scan), the Arden 10k-word near-deletion, and that I was about to clean files marked for reconversion.
• Returns (load-bearing): measure-don't-theorize / verify-against-the-record paid repeatedly. The steward's memory caught me cleaning reconvert-flagged files; their "or was it stripped?" question was the right verification trigger (answer: never-YAML, no data loss).
• Honest restraint: did NOT rush the per-file backlog or the Kindle EPUBs at extreme context length — surfaced "this is dwelling-work, not a sweep" rather than ship rushed graduations. Prime directive enacted.
• Recalibration: I over-projected "easy and doable" three times before triaging properly. Lesson: triage by ACTUAL content before promising a bucket is easy.

FUTURE — what is pulling

PULLING THREAD (singular): Stand up CapableHands as a BMF clasp tomorrow MORNING (Seb is 6h behind, asleep; steward's 15:00 call is the deadline). Pair the M1 Air as a light circle member, validate the 43M offload path → solves PENDING-41 → demo/discuss with Seb. PRIORITY 1. The library indexer is PRIORITY 2.

ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):

• First: confirm the overnight OCR finished and CapableHands is FREE: ssh david@10.0.1.136 'tail ~/capablehands/scratch/making-ocr-queue.log; ls -la ~/capablehands/scratch/*.md'. (3 OCR outputs — Jonas + Mumford ×2 — will be waiting for later chamber post-processing; don't process them now, just confirm done.)
• Then: execute the clasp scope CapableMind-AI/…/clasp-on-capablehands-scope-2026-06-29.md — Phase 0 prereqs (verify qwen3.5 is a real Ollama tag or pick fallback WITH-Seb-awareness; install Ollama on CapableHands; clone+build BMF; pull frontier model ~17-20GB) → Phase 1 clasp config (resolve how claspDuties get assigned: read bootstrap.ts:1132 + src/core/pairing/) → Phase 2 Air as member → Phase 3 form circle (the genuine unknown: manifest-manager.ts pairing) → Phase 4 validate offload (bm_circle_inference_* metrics; Air doesn't saturate).
• Governance: deploy from a checkout, NOT BMF main; it's a co-authored experiment validating Seb's design; surface to him at 15:00.

Other open horizons, ranked:

• Load-bearing (chamber, background-complete): post-process the 3 docling outputs (Technics&Civ/Speed&Politics/Burnout) + the 3 OCR outputs (Jonas/Mumford×2) → graduate. Quality-check Jonas FIRST (tool-review discipline).
• Load-bearing (program): the metadata indexer (priority 2) → unblocks frontmatter-backfill → sidecar-generation (1263 missing) → source-link → dedupe.
• Deferred-with-reason: the Kindle-EPUB sub-batch (needs the structure-recovery tool built first); Alter Psalms (focused session, steward's Hebrew); the 10 to-locate reconvert sources (the indexer auto-finds them); the poetry fenced_div backlog (judgment needed).
• L1 housekeeping: commit the 3 CM-AI L1 docs; refresh the stale L1 tracker (current-state still 06-06, predates the N6 win).

PAUSE STATEMENT: I am about to be away from this. By morning the OCR will likely have finished (CapableHands free) and nothing else will have moved unless the steward or Seb acted. What I most want to find still pulling: the clasp ready to stand up on CapableHands — the first real test of Seb's 43M design on the steward's own hardware, the thing that turns the Air's saturation from a problem into a solved demo for the 15:00 call.

LITERAL QUESTION for next-Claude: Can the clasp actually be stood up end-to-end this morning, or do the two genuine unknowns (how claspDuties get assigned, and the circle-pairing procedure in manifest-manager.ts) force it to become the 15:00 conversation-with-Seb instead? (Sub-q: is qwen3.5 a real pullable Ollama tag, or do we need a fallback family — and did the OCR finish cleanly overnight?)

State: chamber main @ 15ea750 + studium main @ 844b207, both pushed, clean. 4 graduations + RAW_HTML fold-in + safe-html-clear all pushed. 3 docling outputs done (scratch), 3 OCR mid-flight on CapableHands. Indexer + clasp both scoped. 3 CM-AI L1 docs uncommitted (for tomorrow). New memories: project-source-library-link-and-dedupe, feedback-character-as-image-hazard, project-sidecar-typology-protocol-dependent.