dotfiles/claude/memory/project-chamber-typography-mining-plan-2026-05-15.md

name, description, type, originSessionId, permalink

name	description	type	originSessionId	permalink
Chamber Typography Mining Plan — 2026-05-15 → next session	Detailed execution plan for mining a priority voice subset from chamber-library into a clean palace, enabling sustained corpus consultation for ARC work in flight (multilingual typography spec including new English §3, AldineXXI page rewrite, attention page rewrite, Print posture refining). Self-contained for either /wrap-up + /wake-up or clear-context execution.	project	a5214e9b-ee20-435a-8333-f7e645f8ede0	claude-memory/project-chamber-typography-mining-plan-2026-05-15

Chamber Typography Mining Plan — 2026-05-15

Why this plan exists

Today's session diagnosed the death of the chamber-library bge-m3 mine: 1,290,916 drawers in SQLite, HNSW only 27% indexed (346,390), 37 corruption snapshots accumulated. Root cause: HNSW batch_size:50000 + sync_threshold:50000 set in the collection config, combined with bge-m3 on MPS being too slow to flush 50k-vector batches before the next mega-write arrived. Lock fix #1162 alone did not prevent it.

The steward's ARC work in flight needs corpus consultation: multilingual typography spec refinement (FR §1 + ES §2 v0.2; new English §3 to be added), AldineXXI page rewrite (8-voice chavruta from 2026-05-10), attention page rewrite, Print posture refining. All sustained-consultation-shaped, not one-shot lookup work. Direct file reads would tax out by the third lookup.

The decision: mine a critical voice subset (~15–20 source files; ~50–100k drawers) into a separate clean palace, with HNSW config fixed at creation. Vault gets its own palace alongside. Existing broken chamber palace moves aside (preserved as Curiosity Engine Stage 1 substrate candidate if Jurist authorizes).

Inherited state (read before executing)

• MemPalace branch: ~/_Dev/mempalace/ on local/bge-m3-on-3.3.5. Three local commits: bge-m3 swap, embedding_model stamping, MAX_CHUNKS_PER_FILE 500→50_000. Branch stays; only HNSW config changes.
• Broken chamber palace: ~/.mempalace/palace/ — 1.29M drawers, HNSW broken, 11 .corrupt-* + 26 .drift-* snapshots. DO NOT TOUCH except to move aside.
• Hooks state: DISABLED at ~/.claude/settings.json. Backup at ~/.claude/settings.json.backup-2026-05-14-pre-mine-completion. Restore ONLY AFTER pointing at the new vault palace (NOT the broken chamber palace).
• Curiosity Engine: separate project, repo at github.com/davidglidden/curiosity-engine (private), seed brief committed (e13cfad), awaiting Jurist review. The chamber-typography subset mined per THIS plan serves a dual role: immediate ARC consultation AND eventual Stage 1 test substrate for Curiosity Engine.
• Chamber-cruft state: per project-chamber-cruft-restoration.md — 399 files total: 162 clean / 49 near-clean / 124 CLEANUP / 62 severe RECONVERT. Priority voices for this plan are unknown-state until audited (Phase 1b below). Bachelard explicitly named as severe-RECONVERT in 5 books.

Phase 1 — Diagnostics (mandatory before any mining)

1a. Decouple MAX_CHUNKS_PER_FILE from HNSW batch_size (~15 min)

Important context (steward correction 2026-05-15): MAX_CHUNKS_PER_FILE = 50_000 is DELIBERATE — local commit 604df3b raised it from 500 because Loeb bilingual files (long scholarly bilingual editions) were being silently truncated/skipped at the lower limit. That dial must stay at 50_000. Lowering it would re-break Loeb ingestion.

The corruption-causing dial is the HNSW batch_size:50000 in collections.config_json_str, which appears to share the number (probably coupled to MAX_CHUNKS_PER_FILE in code). These need to be decoupled, not both lowered.

Goal of the investigation: find where HNSW batch_size and sync_threshold are set when a collection is created, identify if/how they're coupled to MAX_CHUNKS_PER_FILE, and decouple them. Target values: HNSW batch_size=1000, sync_threshold=1000 (sane for bge-m3 MPS throughput) while MAX_CHUNKS_PER_FILE stays at 50_000.

Search pattern:

cd ~/_Dev/mempalace
grep -rn "batch_size" mempalace/ | grep -iE "50000|50_000|MAX_CHUNKS|hnsw"
grep -rn "sync_threshold" mempalace/
grep -rn "MAX_CHUNKS_PER_FILE" mempalace/
grep -rn "hnsw" mempalace/ | grep -iE "config|batch|sync|create"

Two plausible coupling shapes:

• (a) Collection-creation code sets hnsw.batch_size = MAX_CHUNKS_PER_FILE directly (single-source coupling). Fix: decouple — set HNSW batch_size from its own constant (e.g., HNSW_BATCH_SIZE = 1000).
• (b) Separate but happens to share the value. Fix: just lower HNSW batch_size in whichever file sets it.

Fix path: env var override at mine time if possible (least invasive). If env var doesn't exist, one-line local commit decoupling the constants.

Validation: after fix, create a tiny test palace and inspect collections.config_json_str — should show hnsw.batch_size:1000 AND mining a long Loeb file should still produce >500 chunks (i.e., MAX_CHUNKS_PER_FILE still effectively 50_000).

1b. Cruft audit for priority voices (~15 min)

Cross-reference the priority voice list (Phase 4 below) against project-chamber-cruft-restoration.md. For each priority voice, determine: CLEAN / NEAR-CLEAN / CLEANUP / RECONVERT.

Specifically check the load-bearing voices not used in yesterday's sessions (which had to be clean to be useful):

Voice	Books	Used recently	Cruft state
Bringhurst	Elements of Typographic Style	yes (AldineXXI)	likely clean
Hochuli	Detail in Typography	yes	likely clean
Tschichold	Form of the Book, New Typography	yes	unknown
Lacroux	Orthotypographie v1, v2	yes (multilingual spec)	clean (operative yesterday)
Lexique Imprimerie nationale	—	yes	clean (operative yesterday)
Sousa	Ortografía y ortotipografía	yes	clean (operative yesterday)
Butterick	Practical Typography	—	unknown
Hart's Rules	—	—	unknown
Chicago Manual of Style	—	—	may not be in library
Lupton	Thinking with Type	yes (chavruta)	unknown
Manutius / Margolis	—	yes (chavruta)	unknown
Alexander	Timeless Way, Pattern Language	yes (chavruta)	unknown
Bachelard	Poetics of Space, Earth and Reveries	yes (chavruta)	RECONVERT (5 books flagged severe)
Calvino	Six Memos	yes (chavruta)	unknown
Müller-Brockmann	Grid Systems	—	unknown
de Hamel	—	—	unknown
Saenger	Space Between Words	—	unknown
Williamson	—	—	unknown
Davies	—	—	unknown

1c. Cruft audit confirms cleaning targets (decisions absorbed 2026-05-15)

Scope decision (steward, 2026-05-15): option (b) — clean one representative book per foundational voice in the cruft bucket. Minimum-viable-clean per voice; not the full chamber-library presence per voice.

Foundational voices to be cleaned, with per-voice book choice locked:

• Alexander — Pattern Language (severe RECONVERT)
• Bachelard — Poetics of Space (severe RECONVERT, multiple Bachelard books flagged)
• Heidegger — Poetry, Language, Thought (severe RECONVERT)

Foundational voices to be audited (state unknown; clean if needed):

• Berger — Ways of Seeing
• Sennett — The Craftsman
• Vico — New Science (or selected passages)
• Leopardi — Zibaldone (full text is ~2,500 pages; selected portions likely)
• Harrison — The Dominion of the Dead (likely clean — ARC uses Harrison heavily)

Any of these found in CLEANUP / RECONVERT bucket during audit gets the same Alexandrian per-file treatment (Phase 2). If clean as-is, skip to mining.

1d. Attention page voices (decisions absorbed 2026-05-15)

Confirmed by steward:

• Carruthers — The Book of Memory — memory + attention + medieval rhetoric
• Weil — Gravity and Grace — attention as moral act

Two voices. Sufficient for chavruta-shaped tension.

Phase 2 — Cleaning of foundational voices (~3–5 days, parallelisable across short sessions)

Per-voice cleaning per the 1c decision. For each foundational voice in CLEANUP / RECONVERT bucket, produce a clean version of the chosen representative book using:

• ~/_Dev/chamber-library/scripts/repair_epub_headings.py (heading promotion via TOC) — for scholarly EPUBs
• pandoc -t gfm-raw_html (suppress HTML residue) — standard EPUB pipeline
• Alexandrian per-file craft standard — verify chapter integrity, footnote placement, no cruft tokens in body, proper section structure
• Filing discipline: EPUBs stay in ~/Documents/__Chamber Source EPUBs/; cleaned MDs land in ~/_Dev/chamber-library/converted_texts/

Per-book estimated effort (varies by source quality):

• Pattern Language (Alexander) — known severe; ~half-day
• Poetics of Space (Bachelard) — known severe; ~half-day
• Poetry, Language, Thought (Heidegger) — known severe; ~half-day
• Berger / Sennett / Vico / Leopardi / Harrison — audit first; clean as needed (variable, ~half-day each if needed)

Discipline reminders during cleaning (from previous chamber-cruft work):

• Each file gets its own attention; no batch automation
• Verify source-file integrity by inspection, not by tool output alone (per feedback-pandoc-epub-suppress-raw-html.md)
• If a cleaning attempt fails (source too damaged, OCR errors, scanned-image content), surface that — don't paper over

Stop condition for Phase 2: all 8 foundational voices represented by one clean book + any typography masters or multilingual / attention voices found needing cleaning during Phase 1b audit.

Parallelisation note: cleaning is independent per-file. Multiple short sessions can advance Phase 2 without coordination overhead. Phases 3+ wait for Phase 2 to fully complete.

---

Phase 3 — Setup (~15 min total)

3a. Move broken chamber palace aside

mv ~/.mempalace/palace ~/.mempalace/palace.broken-chamber-2026-05

Preserves the SQLite of 1.29M drawers. If Curiosity Engine ever wants to test repair --mode from-sqlite on this substrate, the data is available. Steward's call to reclaim disk space later.

3b. Hooks restoration pointed at vault palace

cp ~/.claude/settings.json.backup-2026-05-14-pre-mine-completion ~/.claude/settings.json

Then edit settings.json to set MEMPALACE_PALACE_PATH (or equivalent env var) for the hook invocations to ~/.mempalace/palace-vault/. CRITICAL: do NOT let hooks fire against the broken chamber palace or against an empty default.

Alternative: rename whichever palace is to be the default to ~/.mempalace/palace/. The vault palace is the right default since it's where session/diary writes go.

3c. Apply Phase 1a config fix

Per Phase 1a finding — env var or local commit. Verify with a tiny test palace creation that config_json_str shows sane values.

Phase 4 — Vault re-mine (~15–30 min)

Vault path: /Users/davidglidden/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch Git mirror: /Users/davidglidden/_Dev/david-root-and-branch-vault-git

Use git mirror to avoid iCloud sync interference during mine. Mining command roughly:

mempalace --palace ~/.mempalace/palace-vault mine ~/_Dev/david-root-and-branch-vault-git --mode convos --wing vault

(verify exact flags against mempalace mine --help at execution time)

Validation:

• mempalace --palace ~/.mempalace/palace-vault status shows expected drawer count
• Recall test on a known-recent note returns it

Phase 5 — Chamber-typography subset mine (~1–2 hours wall-clock; runs in background)

Mine the 20-book locked corpus into ~/.mempalace/palace-chamber-typography/.

The 8 foundational voices (per ARC colophon Lineage, all must be present):

• Alexander — Pattern Language (cleaned in Phase 2)
• Bachelard — Poetics of Space (cleaned in Phase 2)
• Berger — Ways of Seeing (audit; clean if needed)
• Sennett — The Craftsman (audit; clean if needed)
• Vico — New Science or selected (audit; clean if needed)
• Leopardi — Zibaldone selected portions (audit; clean if needed)
• Harrison — The Dominion of the Dead (audit; likely clean)
• Heidegger — Poetry, Language, Thought (cleaned in Phase 2)

Typography masters (5 books, for AldineXXI / Print posture / cross-task):

• Bringhurst — Elements of Typographic Style
• Tschichold — Form of the Book
• Hochuli — Detail in Typography
• Lupton — Thinking with Type
• Calvino — Six Memos (literary register for AldineXXI; distinct contribution from Bachelard)

Multilingual authorities (5 books):

• Lacroux — Orthotypographie v1
• Lacroux — Orthotypographie v2
• Lexique de l'Imprimerie nationale
• Sousa — Ortografía y ortotipografía del español
• Butterick — Practical Typography (for new English §3)

Attention voices (2 books):

• Carruthers — The Book of Memory
• Weil — Gravity and Grace

Total: 20 books. Cut from earlier lists: Manutius (covered by Bringhurst/Tschichold/Hochuli's treatment of typography history); Hart's Rules (Butterick + Bringhurst cover modern English); Chicago Manual (encyclopedic, overkill for our scope); Müller-Brockmann/de Hamel/Saenger/Williamson/Davies (typography breadth; defer — not blocking ARC-up-to-vignette work).

Voice metadata: each source mined with voice (e.g., "Bachelard") and work (e.g., "Poetics of Space") fields in frontmatter or per-source metadata so --source or --wing scoping works. Verify mempalace's source-level scoping at execution.

Approximate scale: 20 books × avg 3–5k drawers each (with sensible chunking; no Loeb files in this subset so MAX_CHUNKS_PER_FILE will not actually be hit) = 60–100k drawers. At bge-m3 MPS ~50k/h with FIXED HNSW config (per Phase 1a): 1–2 hours wall-clock.

Phase 6 — Validation (~15 min)

Three test queries:

1. Cross-language: "italic for emphasis" — should surface Lexique (FR) AND Bringhurst (EN) within top 10 results. Cross-language retrieval was confirmed at cosine 0.64 on 2026-05-11 PR #442 test.
2. Specific concept: "Van de Graaf canon" or "type-area construction" — should surface Hochuli, Bringhurst, Tschichold; one or two strong hits per source, not 25 from one book.
3. Cross-voice synthesis: "voice and form" or "reading as construction" — should surface multiple voices (Bachelard, Carruthers if present, Bringhurst, etc.), not one source dominating.

Failure modes to watch:

• Density-dilution returning (one source dominates 70%+ of hits)
• Empty results on terms that should be present
• High-cosine matches on cruft tokens (should not appear if Phase 1c done properly)

If validation fails: DIAGNOSE, do not rationalize. The failure is information.

Phase 7 — ARC work begins

With the chamber-typography palace consulted on demand:

mempalace --palace ~/.mempalace/palace-chamber-typography search "query" --source bringhurst

(verify flag syntax against actual CLI)

Suggested task order:

1. Print posture refining — already lived-with for ~24h; iterating on feel. Light first task.
2. Multilingual typography spec refinement — FR §1 + ES §2 to v1.0 LOCKED + new English §3 drafted to v0.1 then v0.2. ~3–5h sustained.
3. AldineXXI page rewrite resumption — masters' reading session from 2026-05-10 pulling thread.
4. Attention page rewrite — after voices named.

Risks (carry forward as discipline)

• Naive mining: Phase 3 or 4 without Phase 1 = repeat corruption cycle. The discipline is: investigation before action.
• Hook restoration to wrong palace: hooks pointing at the broken chamber palace would write diary entries into a dead substrate. Verify hook palace path before restoring.
• Voice list overreach: 15–20 sources is sufficient. Resist the urge to "mine everything that might be useful." More is worse here — density-dilution risk grows.
• Cruft contamination: mining a source still in CLEANUP/RECONVERT will reproduce density-dilution for that voice. Either clean or skip; don't mine dirty.
• Curiosity Engine context drift: this subset palace has a dual role. Don't lose sight of (a) immediate ARC consultation, (b) Curiosity Engine Stage 1 test substrate.

Success criteria

• Clean palace at ~/.mempalace/palace-chamber-typography/
• Voice-scoped recall works
• Density-dilution not visible at this scale
• ARC tasks (multilingual spec including new English §3, AldineXXI, attention, Print posture) can proceed with corpus consultation
• Vault palace also clean and hook-fed
• Broken chamber palace preserved (not deleted) for Curiosity Engine consideration

What this plan does NOT cover

• Curiosity Engine implementation (gated on Jurist review of seed brief)
• Deep landscape audit (Jurist's territory per seed brief §11.7)
• Full chamber-library cleanup (multi-day; separate workflow)
• L1 work (check gh pr view 172 / gh issue view 165 / gh issue view 167 in transition; pick up only if Seb has engaged)

Discipline notes for the executor in the next session

• Symmetria init at session start if non-trivial (this plan is non-trivial; do it).
• Drift pattern to watch: warning-in-context-still-launched — Phase 1 is the warning; do not skip to Phase 3 because mining feels productive. Investigation is productive too.
• Drift pattern to watch: forcing-the-plan-onto-changed-evidence — if Phase 1b finds the cruft state is much worse than expected, the plan may need restructuring. Surface, don't paper over.
• Honest empty: if a phase produces a finding that invalidates a downstream phase, say so and stop. The steward authorizes restructure; the executor doesn't barrel through.