dotfiles/claude/memory/project-mempalace-mps-transition-pending.md

---
permalink: claude-memory/project-mempalace-mps-transition-pending
superseded_by: project-mempalace-winddown.md
superseded_on: 2026-08-17
---
> **SUPERSEDED 2026-08-17.** A paused bge-m3/MPS mine on a palace that was wiped and then retired. The backup paths named below may still exist on disk and are the only part worth acting on.
>
> Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the
> status below as live.


---
name: MemPalace bge-m3 + MPS transition — paused mid-flight; resume in fresh session
description: Chamber-library mine with bge-m3 was running on CPU and projected to take 4-5 days. Plan: enable Apple Silicon Metal (MPS) acceleration via chroma.py patch addition (~5-10x speedup, finish in hours not days). Processes already killed and palace wiped clean to skeleton; ready to resume. Independent BMF audit thread also surfaced.
type: project
originSessionId: 17dee884-74bf-43e7-9b6a-44bc82b97303
---
# MemPalace bge-m3 + MPS transition — paused

Started 2026-04-30 evening, paused 2026-05-01 morning after diagnosing the speed problem. Steward chose fresh session for the resume rather than continuing in a long context.

## Current state at pause (2026-05-01 ~10:30 AM Barcelona)

- **Chamber mine killed** — PID 9199 (was 30/371 files in 9 hours under bge-m3 CPU). Projected 4-5 days at that rate; abandoned.
- **BMF service killed** — PID 1043. Was at 224% CPU for 14+ hours. Investigation surfaced a Lance store pathology (1,349 index segments, 25,449+ transactions, no compaction). **NOT issue #10** (steward 2026-05-01: "BMF issue #10 was resolved ages ago"). This is a fresh pathology and needs a deliberate audit session with steward + Seb. Lance is resumable — restart cleanly when audit happens.
- **Palace wiped to skeleton** — `~/.mempalace/palace/` is the 168 KB fresh skeleton from `palace.skeleton-backup-2026-04-30`. Both prior backups preserved:
  - `palace.pre-bge-m3-2026-04-30` (956 MB; full all-MiniLM state under 263k drawers; rollback target if bge-m3 ever proves wrong)
  - `palace.crashed-2026-04-30` (141 MB sqlite; forensic value if anyone wants to inspect the original crashed state)
- **chroma.py patch in place** — bge-m3 model selection (lines ~26-46). Just needs ONE more line: `device="mps"`.
- **Global entity registry clean** — `~/.mempalace/known_entities.json` has the canonical 21-person list (Alexander, Bachelard, Berger, Bringhurst, Derrida, Harrison, Heidegger, Leopardi, Manutius, Norris, Sennett, Stevens, Vico, Warde, etc.). The per-repo `chamber-library/entities.json` is still noisy from yesterday's `init --llm --yes` but mining doesn't read that file (verified: miner.py loads `~/.mempalace/known_entities.json` only).

## Decisions locked

1. **Stay with bge-m3** (not switch to a faster English-only model). Chamber library is overwhelmingly English but contains *critical* French texts and a couple in German. Quality for citation-grade Chamber retrieval is non-negotiable; multilingual coverage matters for the French/German passages.
2. **Add MPS (Metal Performance Shaders) acceleration.** Apple Silicon GPU sits idle; sentence-transformers supports `device='mps'` natively. Same model, same outputs, runs on Metal cores. ~5-10x speedup expected for 1024-dim transformer embedding work.
3. **Mine `converted_texts/` only**, not the whole repo. `scripts/` is stale per steward 2026-05-01. Use `--wing chamber_library` explicitly (otherwise the wing defaults to directory name).
4. **mempalace.yaml stale but ignorable.** The yaml at `~/_Dev/chamber-library/mempalace.yaml` lists rooms that no longer exist on disk (observer_problem_sources, derrida, jaccottet:handke:lichtenberg). Mining `converted_texts/` directly sidesteps the yaml entirely; mempalace will use a single "general" room. Sub-room granularity (collections, dialogues, indices, traditions as separate rooms) can be added later via a proper yaml + re-roomification — does not require re-embedding.

## Resume plan — 5 steps for the fresh session

1. **Verify state** — `ps aux | grep -iE "mempalace|node.*Better"` should show nothing relevant. Palace should be the 168K skeleton at `~/.mempalace/palace/`.
2. **Edit `chroma.py`** — file: `/Users/davidglidden/_Dev/mempalace/mempalace/backends/chroma.py`. The `_mempalace_embedding_fn()` function around line 38-43 currently reads:
   ```python
   def _mempalace_embedding_fn():
       global _embedding_fn_cache
       if _embedding_fn_cache is None:
           from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
           _embedding_fn_cache = SentenceTransformerEmbeddingFunction(model_name=_MEMPALACE_EMBED_MODEL)
       return _embedding_fn_cache
   ```
   Add the `device="mps"` parameter:
   ```python
   _embedding_fn_cache = SentenceTransformerEmbeddingFunction(
       model_name=_MEMPALACE_EMBED_MODEL,
       device="mps",
   )
   ```
3. **Sanity test MPS** — quick script to verify bge-m3 loads on Metal and produces 1024-dim output:
   ```bash
   ~/_Dev/mempalace/.venv/bin/python -c "
   import time, torch
   print('mps available:', torch.backends.mps.is_available())
   from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
   t0 = time.time()
   ef = SentenceTransformerEmbeddingFunction(model_name='BAAI/bge-m3', device='mps')
   print(f'load: {time.time()-t0:.1f}s')
   t0 = time.time()
   v = ef(['hello', 'bonjour le monde', 'guten tag'])
   print(f'embed 3: {time.time()-t0:.2f}s, dim={len(v[0])}')
   "
   ```
   Expected: load ~10s (model cached), embed 3 sentences ~0.1-0.5s, dim=1024.
4. **Start mine** — same form as before:
   ```bash
   nohup ~/_Dev/mempalace/.venv/bin/mempalace mine ~/_Dev/chamber-library/converted_texts --wing chamber_library > /tmp/mine-chamber-library-$(date +%Y-%m-%d).log 2>&1 < /dev/null & disown
   ```
5. **Verify GPU engagement** — within 30 seconds:
   - Activity Monitor → GPU view should show GPU utilization
   - Log shows "Loading weights" then file processing
   - Estimated rate at MPS: ~16-30 files/hr (vs 3.3 files/hr on CPU)
   - Total chamber-library: ~12-24 hours (vs 4-5 days on CPU)

## After chamber-library completes

Three other wings need re-mining under bge-m3 + MPS, in the steward's preferred order:
- **Phase 4b** — Obsidian vault (was 112,558 drawers under all-MiniLM-L6-v2). Path: `~/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch`. `--wing obsidian-vault`.
- **Phase 4c** — Claude transcripts. Path: `~/.claude/projects/-Users-davidglidden`. `--mode convos`. Was 12,688 drawers. (Note: a `Stop` hook auto-fires `mempalace mine` on this directory — saw it spawn an unauthorized parallel mine 2026-04-30 night; if the hook is still active, it'll do this work automatically when sessions end. Otherwise queue manually.)
- **Phase 4d** — CapableMind thinking docs. Path: `~/_Dev/CapableMind-AI/docs/thinking/David/`. `--wing capablemind-thinking`. Was 6,137 drawers.

Likely 1-2 days total for all three under MPS. Could run sequentially overnight on consecutive nights.

## Independent thread surfaced — BMF L1 audit

Killing BMF (PID 1043) revealed a real L1 pathology that needs deliberate attention:

- **1,349 index segments** in `~/.capablemind/data/mindfabric-00/lance/vector_chunks.lance/_indices/` (healthy Lance stores have 1-10)
- **25,449+ transactions** in the same store
- 14+ hours of sustained 224% CPU consuming all those segments per query
- Data store itself is small (139 MB) — the bloat is metadata/segment count, not data volume

This is **not** issue #10 (resolved). It's a fresh pathology — likely Lance compaction never ran, or a write-amplification loop in BMF's ingestion path creates a new index segment per drawer/batch.

Belongs to: a separate L1 audit session, steward + Seb territory. Lance state is preserved (BMF's killed but data intact); when audit happens, `pkill node` cleanup is already done.

Reference forensic notes from prior L1 audit: `~/_Dev/CapableMind-AI/docs/thinking/David/l1-reliability/` (`l1-deep-audit-plan-2026-04-19.md`, `l1-ingestion-forensic-2026-04-19.md`, `l1-recall-quality-baseline-2026-04-19.md`).

## Why this transition felt long

Steward 2026-05-01: *"I can see how mempalace helps us immensely and want to finish this update transition which I almost regret doing — I didn't think it would take days to do."*

Honest accounting:
- Power failure 2026-04-30 mid-stream cost a full mine + caused the 217 GB HNSW corruption requiring full reset.
- bge-m3 on CPU is genuinely 5-10x slower than was reasonable to expect for this corpus size. CPU choice was a default-not-a-decision; should have proposed MPS from the start.
- Multiple parallel processes (auto-Stop-hook transcripts mine + BMF) competed for CPU and confused diagnosis.
- Each diagnostic round added time without throughput.

After the MPS switch the transition should feel proportionate to its value.

## Pulling thread for the fresh session

**Apply the `device="mps"` patch and restart the chamber-library mine; everything else (palace state, kills, scope decision, model decision) is already done.** Five steps from "open a fresh session" to "mine running on Metal."