dotfiles/claude/memory/project-mempalace-mps-transition-pending.md

permalink, superseded_by, superseded_on

permalink	superseded_by	superseded_on
claude-memory/project-mempalace-mps-transition-pending	project-mempalace-winddown.md	2026-08-17

SUPERSEDED 2026-08-17. A paused bge-m3/MPS mine on a palace that was wiped and then retired. The backup paths named below may still exist on disk and are the only part worth acting on.

Current record: project-mempalace-winddown.md. Kept for detail and provenance — do not read the status below as live.

---

name: MemPalace bge-m3 + MPS transition — paused mid-flight; resume in fresh session description: Chamber-library mine with bge-m3 was running on CPU and projected to take 4-5 days. Plan: enable Apple Silicon Metal (MPS) acceleration via chroma.py patch addition (~5-10x speedup, finish in hours not days). Processes already killed and palace wiped clean to skeleton; ready to resume. Independent BMF audit thread also surfaced. type: project originSessionId: 17dee884-74bf-43e7-9b6a-44bc82b97303

MemPalace bge-m3 + MPS transition — paused

Started 2026-04-30 evening, paused 2026-05-01 morning after diagnosing the speed problem. Steward chose fresh session for the resume rather than continuing in a long context.

Current state at pause (2026-05-01 ~10:30 AM Barcelona)

• Chamber mine killed — PID 9199 (was 30/371 files in 9 hours under bge-m3 CPU). Projected 4-5 days at that rate; abandoned.
• BMF service killed — PID 1043. Was at 224% CPU for 14+ hours. Investigation surfaced a Lance store pathology (1,349 index segments, 25,449+ transactions, no compaction). NOT issue #10 (steward 2026-05-01: "BMF issue #10 was resolved ages ago"). This is a fresh pathology and needs a deliberate audit session with steward + Seb. Lance is resumable — restart cleanly when audit happens.
• Palace wiped to skeleton — ~/.mempalace/palace/ is the 168 KB fresh skeleton from palace.skeleton-backup-2026-04-30. Both prior backups preserved:
  • palace.pre-bge-m3-2026-04-30 (956 MB; full all-MiniLM state under 263k drawers; rollback target if bge-m3 ever proves wrong)
  • palace.crashed-2026-04-30 (141 MB sqlite; forensic value if anyone wants to inspect the original crashed state)
• chroma.py patch in place — bge-m3 model selection (lines ~26-46). Just needs ONE more line: device="mps".
• Global entity registry clean — ~/.mempalace/known_entities.json has the canonical 21-person list (Alexander, Bachelard, Berger, Bringhurst, Derrida, Harrison, Heidegger, Leopardi, Manutius, Norris, Sennett, Stevens, Vico, Warde, etc.). The per-repo chamber-library/entities.json is still noisy from yesterday's init --llm --yes but mining doesn't read that file (verified: miner.py loads ~/.mempalace/known_entities.json only).

Decisions locked

1. Stay with bge-m3 (not switch to a faster English-only model). Chamber library is overwhelmingly English but contains critical French texts and a couple in German. Quality for citation-grade Chamber retrieval is non-negotiable; multilingual coverage matters for the French/German passages.
2. Add MPS (Metal Performance Shaders) acceleration. Apple Silicon GPU sits idle; sentence-transformers supports device='mps' natively. Same model, same outputs, runs on Metal cores. ~5-10x speedup expected for 1024-dim transformer embedding work.
3. Mine converted_texts/ only, not the whole repo. scripts/ is stale per steward 2026-05-01. Use --wing chamber_library explicitly (otherwise the wing defaults to directory name).
4. mempalace.yaml stale but ignorable. The yaml at ~/_Dev/chamber-library/mempalace.yaml lists rooms that no longer exist on disk (observer_problem_sources, derrida, jaccottet:handke:lichtenberg). Mining converted_texts/ directly sidesteps the yaml entirely; mempalace will use a single "general" room. Sub-room granularity (collections, dialogues, indices, traditions as separate rooms) can be added later via a proper yaml + re-roomification — does not require re-embedding.

Resume plan — 5 steps for the fresh session

1. Verify state — ps aux | grep -iE "mempalace|node.*Better" should show nothing relevant. Palace should be the 168K skeleton at ~/.mempalace/palace/.
2. Edit chroma.py — file: /Users/davidglidden/_Dev/mempalace/mempalace/backends/chroma.py. The _mempalace_embedding_fn() function around line 38-43 currently reads:

   def _mempalace_embedding_fn():
       global _embedding_fn_cache
       if _embedding_fn_cache is None:
           from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
           _embedding_fn_cache = SentenceTransformerEmbeddingFunction(model_name=_MEMPALACE_EMBED_MODEL)
       return _embedding_fn_cache
   

   Add the device="mps" parameter:

   _embedding_fn_cache = SentenceTransformerEmbeddingFunction(
       model_name=_MEMPALACE_EMBED_MODEL,
       device="mps",
   )
   

3. Sanity test MPS — quick script to verify bge-m3 loads on Metal and produces 1024-dim output:

   ~/_Dev/mempalace/.venv/bin/python -c "
   import time, torch
   print('mps available:', torch.backends.mps.is_available())
   from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
   t0 = time.time()
   ef = SentenceTransformerEmbeddingFunction(model_name='BAAI/bge-m3', device='mps')
   print(f'load: {time.time()-t0:.1f}s')
   t0 = time.time()
   v = ef(['hello', 'bonjour le monde', 'guten tag'])
   print(f'embed 3: {time.time()-t0:.2f}s, dim={len(v[0])}')
   "
   

   Expected: load ~10s (model cached), embed 3 sentences ~0.1-0.5s, dim=1024.
4. Start mine — same form as before:

   nohup ~/_Dev/mempalace/.venv/bin/mempalace mine ~/_Dev/chamber-library/converted_texts --wing chamber_library > /tmp/mine-chamber-library-$(date +%Y-%m-%d).log 2>&1 < /dev/null & disown
   

5. Verify GPU engagement — within 30 seconds:
   • Activity Monitor → GPU view should show GPU utilization
   • Log shows "Loading weights" then file processing
   • Estimated rate at MPS: ~16-30 files/hr (vs 3.3 files/hr on CPU)
   • Total chamber-library: ~12-24 hours (vs 4-5 days on CPU)

After chamber-library completes

Three other wings need re-mining under bge-m3 + MPS, in the steward's preferred order:

• Phase 4b — Obsidian vault (was 112,558 drawers under all-MiniLM-L6-v2). Path: ~/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch. --wing obsidian-vault.
• Phase 4c — Claude transcripts. Path: ~/.claude/projects/-Users-davidglidden. --mode convos. Was 12,688 drawers. (Note: a Stop hook auto-fires mempalace mine on this directory — saw it spawn an unauthorized parallel mine 2026-04-30 night; if the hook is still active, it'll do this work automatically when sessions end. Otherwise queue manually.)
• Phase 4d — CapableMind thinking docs. Path: ~/_Dev/CapableMind-AI/docs/thinking/David/. --wing capablemind-thinking. Was 6,137 drawers.

Likely 1-2 days total for all three under MPS. Could run sequentially overnight on consecutive nights.

Independent thread surfaced — BMF L1 audit

Killing BMF (PID 1043) revealed a real L1 pathology that needs deliberate attention:

• 1,349 index segments in ~/.capablemind/data/mindfabric-00/lance/vector_chunks.lance/_indices/ (healthy Lance stores have 1-10)
• 25,449+ transactions in the same store
• 14+ hours of sustained 224% CPU consuming all those segments per query
• Data store itself is small (139 MB) — the bloat is metadata/segment count, not data volume

This is not issue #10 (resolved). It's a fresh pathology — likely Lance compaction never ran, or a write-amplification loop in BMF's ingestion path creates a new index segment per drawer/batch.

Belongs to: a separate L1 audit session, steward + Seb territory. Lance state is preserved (BMF's killed but data intact); when audit happens, pkill node cleanup is already done.

Reference forensic notes from prior L1 audit: ~/_Dev/CapableMind-AI/docs/thinking/David/l1-reliability/ (l1-deep-audit-plan-2026-04-19.md, l1-ingestion-forensic-2026-04-19.md, l1-recall-quality-baseline-2026-04-19.md).

Why this transition felt long

Steward 2026-05-01: "I can see how mempalace helps us immensely and want to finish this update transition which I almost regret doing — I didn't think it would take days to do."

Honest accounting:

• Power failure 2026-04-30 mid-stream cost a full mine + caused the 217 GB HNSW corruption requiring full reset.
• bge-m3 on CPU is genuinely 5-10x slower than was reasonable to expect for this corpus size. CPU choice was a default-not-a-decision; should have proposed MPS from the start.
• Multiple parallel processes (auto-Stop-hook transcripts mine + BMF) competed for CPU and confused diagnosis.
• Each diagnostic round added time without throughput.

After the MPS switch the transition should feel proportionate to its value.

Pulling thread for the fresh session

Apply the device="mps" patch and restart the chamber-library mine; everything else (palace state, kills, scope decision, model decision) is already done. Five steps from "open a fresh session" to "mine running on Metal."