70 lines
10 KiB
Markdown
70 lines
10 KiB
Markdown
---
|
||
name: session-2026-06-13-studium-engine-steps-3-4-built-chamber-library-canonical-texts-tier-176-source-graduation
|
||
description: "Enormous build session. Resolved the reading-index home (chamber-library/reading-indices/) + built the canonical_texts tier with a cleanliness-gated 176-source graduation, generated catalogue.yaml source-of-truth, filename canonicalization, pre-commit hook reconciliation. Then Studium engine Steps 3 (chunker) + 4 (store+FTS5+coverage ledger) — both verified (Weber non-leak, deterministic rebuild ID-equality, byte-faithful, paratext non-indexable). 10 commits / 3 repos, all pushed. PULLING THREAD: brief detour to L1 (Seb's reply landed in CapableMind-AI) BEFORE returning to Studium Step 5 (retrieval primitives)."
|
||
metadata:
|
||
node_type: memory
|
||
type: project
|
||
originSessionId: 4d9109ef-7b04-40c9-8ef0-ffb79bf74435
|
||
---
|
||
|
||
# Session 2026-06-13 — Studium Engine Steps 3-4 + chamber-library canonical tier
|
||
|
||
Woke into the Studium thread (resolve reading-index home → commit → Step 3). The session expanded enormously through a chain of steward refinements, each building on the last. Symmetria active (ledger `session-ledger-2026-06-13.md`).
|
||
|
||
## Past — what we did (the arc, in order)
|
||
|
||
**1. Reading-index home + source tier (the morning thread, then it grew).** Steward ruled: cleaned sources graduate from `converted_texts/` (raw inbox) → **`canonical_texts/`** (cleaned+confirmed, engine-eligible) — a visible structural tier, not a flag (his layer/placement instinct, load-bearing again). Named `canonical_texts` (state contrast vs `converted_texts`). Moved Alexander + Mauss in (hashes proven identical — pure path-edit in the manifest). Reading indices → **`chamber-library/reading-indices/`** (sibling, single home); studium-engine derived copies killed; manifest repointed (8 lines).
|
||
|
||
**2. CM-AI reading-indices: SOFT-retire (steward-ruled).** Found the 4 CM-AI indices are referenced by 5 other files (live `voices/*.yaml` config + README links + phase-2 prompts) — a hard delete would break them. Surfaced before acting → steward chose **soft-retire in place**: stamped each `RETIRED — canonical home now chamber-library/...`, references still resolve, frozen prior-art. (david-after-the-reply CM-AI copy is the known-obsolete unparseable original, left as-is.)
|
||
|
||
**3. catalogue.yaml — generated single source of truth.** Found the existing `CHAMBER_CATALOGUE.md` is hand-maintained (drifts). Built `scripts/build_catalogue.py` (filesystem-walked, `--check` drift gate) → `catalogue.yaml` (417 sources, tier, neighborhood, frontmatter-or-null, engine cross-ref). Surfaced honest gaps: 179 sources lack frontmatter.
|
||
|
||
**4. "A folder of already-cleaned files?"** Searched FS + git + MemPalace: **no dedicated folder** — the 2026-05 "in-place cruft cleanup of 37 files" (commit `113d3bb`) was *in-place*, scattered by tradition. Reframed: graduate by **cleanliness NOW**, not cleaning-history (record vs substrate).
|
||
|
||
**5. Cleanliness gate (steward: "update the script to our new definition").** Extended `audit_cruft.py` to the full definition — added the EPUB footnote/TOC/page-image residue signatures (lock-step with `clean_epub_residue.py`), `.txt` coverage, `--clean`/`--summary` modes. It is now THE authoritative cleanliness gate. Census: 415 inbox → **178 cruft-free, 237 dirty**. (Caught my own buggy meta-file filter — `LOG` matched "epistemo**logy**" etc. — corrected to 4 genuine meta files → 174 graduation-eligible source-texts.)
|
||
|
||
**6. Graduation (steward authorized, dry-run first).** Built `scripts/graduate_to_canonical.py` (cruft-gated, dry-run default, canonicalize-name-on-entry, coupled manifest edit for engine sources). Applied: **174 sources graduated** (incl. Harrison — its PDF source was actually clean; manifest path updated). `canonical_texts/` now **176**, `converted_texts/` **241** (237 dirty + 4 meta). Gate 8/8, catalogue `--check` clean.
|
||
|
||
**7. Filename canonicalization (spaces).** Built `scripts/canonicalize_filenames.py` (lowercase-kebab-ASCII, dry-run, git mv, collision+reference guards, engine-aware). Applied **spaces-only (35 files)** per steward; full sweep (196) deferred. Names canonicalize on graduation-entry going forward.
|
||
|
||
**8. Commits + the hook conflict.** Pre-commit hook's 5MB guard blocked the corpus commit — conflicted with the deliberate retire-LFS decision (`0677e8a`, "corpus is plain text in git"). Surfaced (didn't bypass) → steward authorized **exempting corpus text** (`.md/.txt` under canonical_texts/converted_texts) from the guard. **10 commits across 3 repos**, all pushed (chamber-library→Gitea, studium-engine+CM-AI→GitHub).
|
||
|
||
**9. Studium Step 3 — chunker (`engine/chunker.py`).** Section-bounded (one voice/lang/role; paratext withheld), semantic (paragraph/heading, no fixed-N; headings lead content), per-work quality-scored, deterministic `chunk_id = sha256(source_sha256|section_id|char-range)`. **1970 drawers, mean-Q 0.94–1.0.** Verified against substrate: Weber never served as Mauss (0 leak), id-set deterministic, text_original byte-faithful. Fixed an orphan-heading artifact mid-build (degen dropped ~3×).
|
||
|
||
**10. Studium Step 4 — store+index+ledger (`engine/store.py`).** SQLite + FTS5 over text_normalized (diacritic-insensitive) + **coverage ledger** (every line classified served/paratext/apparatus, status, as-of, hash-verified). Routine commands: **`rebuild`** (drop+reconstruct from files, proves chunk-ID-set IDENTICAL — success criterion #1, Class-A non-fatal) + **`verify`** (store↔index↔files divergence probes). Verified: rebuild ID-equal (1970), paratext non-indexable (0 weber drawers), FTS voice-scoping works, every Mauss line classified → silence warrantable. chunks.jsonl + index.db gitignored (derived/disposable).
|
||
|
||
## Present — mood / returns
|
||
- **The session's spine, vindicated again and again: verify against the substrate, not the record.** Every defect (4 reading-index YAMLs last night; my buggy `LOG` meta-filter today; the "Harrison source not cleaned" record contradicted by the cruft audit; the "37 cleaned files" git-archaeology dead end) was caught by checking, not asserting. The chunker/store invariants too — Weber-non-leak, rebuild-ID-equality, byte-faithfulness — *measured*, not trusted.
|
||
- **The steward's layer/placement instincts stayed load-bearing**: canonical_texts as a *visible tier* not a flag; clean-at-source; soft-retire; exempt-corpus-from-guard. Each improved the architecture, not just the output.
|
||
- **Surfaced-not-bypassed held under pressure**: the CM-AI 5-file entanglement, the pre-commit guard conflict — both flagged for steward ruling rather than papered over (contamination directive operative).
|
||
- **Returns logged** (ledger): the buggy-filter false-positive (census-through-a-pattern needs verifying the pattern itself); resisted bulk-moving on the git label.
|
||
|
||
## Future
|
||
|
||
### Pulling thread (singular, but staged)
|
||
**Switch briefly to L1 first — Seb's reply arc landed in CapableMind-AI (`cc25995` reply + `e2e94ab` cover note, now in local main) — THEN return here to Studium Step 5 (retrieval primitives).** The steward explicitly set this order at wrap: a short L1 detour to engage Seb's response, then back to the engine.
|
||
|
||
### Actionable resumption point (as of wrap)
|
||
- **Immediate (L1):** read Seb's reply in `~/_Dev/CapableMind-AI/` — `git show cc25995` and `e2e94ab` (+ `8f76bf2` benchmark-governance v1.1 §4.4). Read the L1 tracker (`project-L1-reliability.md`) for the baton state before composing any response. The memory had it "baton with Seb; do not re-enter until he responds" — he has. Steward decides the response.
|
||
- **Return (Studium Step 5):** `~/_Dev/studium-engine`, `main @ b3274c4` clean+pushed. Steps 0–4 done. Step 5 = **retrieval primitives** (build plan §Step 5): voice-scoped verbatim search returning `(text_original, work, section_id, lines)` constructed FROM retrieval; **honest-empty** (empty result carries its coverage warrant from the ledger, or declares the silence unwarranted); relate-to-thread; reference-drawer "see also"; **selection observability** (retrieved-but-not-surfaced logged — integrity commitment 7). The substrate is ready: `store.py search` is a smoke stub to grow into the real primitive. Success criterion #4 (induced `blocked` unit → unwarranted silence) lands here.
|
||
|
||
### Other open horizons (ranked)
|
||
- **L1 response to Seb** (immediate, steward-led).
|
||
- **Studium Step 5 → 6 (voice-balanced ranker) → 7 (FTS-vs-vector verdict) → 8 (chavruta skill) → 9 (Essay-I re-run).** The application layer on the substrate.
|
||
- **chamber-library: 237 dirty inbox files** — clean + graduate incrementally through the gate (full filename sweep rides along on entry). The 196-file full canonicalization sweep is deferred (3 collisions to hand-resolve).
|
||
- **Sidecar source_path** was trued-up for the 3 graduated (canonical_texts); consider whether sidecars should carry source_path at all (manifest is the path authority — duplication is drift risk).
|
||
- Phase-2 / parked: ARC §VII.f mobile-Safari truth-up; be (Pat WTP).
|
||
|
||
### Pause statement
|
||
I am about to be away. The Studium substrate (Steps 0–4) is built, verified, committed, pushed — nothing at risk; the index is *provably* disposable (rebuild proves it). What I want to find still pulling: **the L1 detour done (Seb engaged) and Studium Step 5 (retrieval primitives) underway over this clean, warranted substrate.** What I do NOT want: to return straight to Step 5 and forget Seb's reply is waiting; or to re-derive the corpus state cold (read catalogue.yaml + the coverage ledger).
|
||
|
||
### Literal question for next-Claude
|
||
**Has Seb's L1 reply been read and responded to (the brief detour the steward set), and is Studium Step 5 — the honest-empty, coverage-warranted retrieval primitive — now being built over the Step-4 substrate?**
|
||
|
||
## Decisions deferred (and why)
|
||
- **Full filename sweep (196 files)** — steward chose spaces-only now; full canonicalization rides graduation incrementally (3 collisions need hand-resolution).
|
||
- **Graduating the 237 dirty files** — gated on per-file cleaning through the now-authoritative `audit_cruft` gate; incremental, not bulk.
|
||
- **The L1 response content** — steward-led; I only surfaced that Seb replied.
|
||
- **Vector sidecar** — deferred to Step 7's FTS-vs-vector verdict by design (FTS-first).
|
||
- **No PENDING items added** — the engine is steward-direct governance (D-1), outside the PENDING/REVIEWED loop.
|