David F GliddenandClaude Opus 5
c2df868104
The 2026-06-06 normalization sweep (3f9a89b, 283 files) prepended a stray permalink-only frontmatter block to 39 memory files, 20 of them session wrap records. Any real frontmatter parser reads the FIRST block, gets `permalink` alone, and never sees name/description/type. The sharp case: `strip_frontmatter` exists specifically to stop thread extraction from matching inside `description:` — its docstring says so. On those 20 wrap records it strips the stray block and hands the real frontmatter back AS BODY, producing exactly the condition it was written to prevent. Repair merges permalink into the real block and drops the duplicate. Nothing retyped: frontmatter and body are carried as slices, body md5 asserted unchanged. TWO DEFECTS OF MY OWN, both found only after claiming success: 1. THREE FILES DAMAGED BY THIS SESSION'S OWN STAMPING COMMIT (ef6fa94). That script located frontmatter with a non-greedy `^---\n.*?\n---\n`, which on a stray-block file matches the STRAY block — so it appended superseded_by to the wrong block and orphaned the real frontmatter into the body. That is the same non-greedy-first-block blindness diagnosed in strip_frontmatter one hour earlier, reproduced in the tool written to clean up after it. And the post-stamp check reported "malformed: none" because it asked "does the file start with frontmatter then a banner" — true for all three — while the claim was "the stamp preserved the record's metadata". The control's subject was adjacent to the claim's, for the sixth time today and the first time in my own verification. Repaired, verified against the pre-stamp file from git: no key lost, prose preserved. 2. REPAIRING 20 APRIL-MAY WRAP RECORDS MOVED THEIR MTIMES TO TODAY, and `sec_pause` picked "newest wrap" by mtime — so the digest promoted an April session to `Last wrap` and lost both the pulling thread and the open question. Caught by the DEGRADED section, which is the honest-degradation mechanism doing its job. `sec_pause` now selects by git add-time, like `wrap_events`. Residual stated in the comment: add-time lags the wrap by hours, so `Last wrap` is an upper bound — wrong by hours where mtime was wrong by months. Verified after: 0 two-block patterns, 0 orphaned frontmatter, 16 stamps intact with name: present, 381 pointers resolve, 0 dead, selftest PASS, drift clean, digest reports 0 degraded sections and the correct last wrap. Not defects: context-discipline.md and relational-gap-analysis.md use the `title:`/`type: note` convention and have no `name:` — flagged by my check, correct as they stand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Y6t6qx7cpaCu5xGdD36u4
118 lines
9.0 KiB
Markdown
118 lines
9.0 KiB
Markdown
---
|
|
name: MemPalace bge-m3 + MPS transition — paused mid-flight; resume in fresh session
|
|
description: Chamber-library mine with bge-m3 was running on CPU and projected to take 4-5 days. Plan: enable Apple Silicon Metal (MPS) acceleration via chroma.py patch addition (~5-10x speedup, finish in hours not days). Processes already killed and palace wiped clean to skeleton; ready to resume. Independent BMF audit thread also surfaced.
|
|
type: project
|
|
originSessionId: 17dee884-74bf-43e7-9b6a-44bc82b97303
|
|
permalink: claude-memory/project-mempalace-mps-transition-pending
|
|
superseded_by: project-mempalace-winddown.md
|
|
superseded_on: 2026-08-17
|
|
---
|
|
> **SUPERSEDED 2026-08-17.** A paused bge-m3/MPS mine on a palace that was wiped and then retired. The backup paths named below may still exist on disk and are the only part worth acting on.
|
|
>
|
|
> Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the
|
|
> status below as live.
|
|
|
|
# MemPalace bge-m3 + MPS transition — paused
|
|
|
|
Started 2026-04-30 evening, paused 2026-05-01 morning after diagnosing the speed problem. Steward chose fresh session for the resume rather than continuing in a long context.
|
|
|
|
## Current state at pause (2026-05-01 ~10:30 AM Barcelona)
|
|
|
|
- **Chamber mine killed** — PID 9199 (was 30/371 files in 9 hours under bge-m3 CPU). Projected 4-5 days at that rate; abandoned.
|
|
- **BMF service killed** — PID 1043. Was at 224% CPU for 14+ hours. Investigation surfaced a Lance store pathology (1,349 index segments, 25,449+ transactions, no compaction). **NOT issue #10** (steward 2026-05-01: "BMF issue #10 was resolved ages ago"). This is a fresh pathology and needs a deliberate audit session with steward + Seb. Lance is resumable — restart cleanly when audit happens.
|
|
- **Palace wiped to skeleton** — `~/.mempalace/palace/` is the 168 KB fresh skeleton from `palace.skeleton-backup-2026-04-30`. Both prior backups preserved:
|
|
- `palace.pre-bge-m3-2026-04-30` (956 MB; full all-MiniLM state under 263k drawers; rollback target if bge-m3 ever proves wrong)
|
|
- `palace.crashed-2026-04-30` (141 MB sqlite; forensic value if anyone wants to inspect the original crashed state)
|
|
- **chroma.py patch in place** — bge-m3 model selection (lines ~26-46). Just needs ONE more line: `device="mps"`.
|
|
- **Global entity registry clean** — `~/.mempalace/known_entities.json` has the canonical 21-person list (Alexander, Bachelard, Berger, Bringhurst, Derrida, Harrison, Heidegger, Leopardi, Manutius, Norris, Sennett, Stevens, Vico, Warde, etc.). The per-repo `chamber-library/entities.json` is still noisy from yesterday's `init --llm --yes` but mining doesn't read that file (verified: miner.py loads `~/.mempalace/known_entities.json` only).
|
|
|
|
## Decisions locked
|
|
|
|
1. **Stay with bge-m3** (not switch to a faster English-only model). Chamber library is overwhelmingly English but contains *critical* French texts and a couple in German. Quality for citation-grade Chamber retrieval is non-negotiable; multilingual coverage matters for the French/German passages.
|
|
2. **Add MPS (Metal Performance Shaders) acceleration.** Apple Silicon GPU sits idle; sentence-transformers supports `device='mps'` natively. Same model, same outputs, runs on Metal cores. ~5-10x speedup expected for 1024-dim transformer embedding work.
|
|
3. **Mine `converted_texts/` only**, not the whole repo. `scripts/` is stale per steward 2026-05-01. Use `--wing chamber_library` explicitly (otherwise the wing defaults to directory name).
|
|
4. **mempalace.yaml stale but ignorable.** The yaml at `~/_Dev/chamber-library/mempalace.yaml` lists rooms that no longer exist on disk (observer_problem_sources, derrida, jaccottet:handke:lichtenberg). Mining `converted_texts/` directly sidesteps the yaml entirely; mempalace will use a single "general" room. Sub-room granularity (collections, dialogues, indices, traditions as separate rooms) can be added later via a proper yaml + re-roomification — does not require re-embedding.
|
|
|
|
## Resume plan — 5 steps for the fresh session
|
|
|
|
1. **Verify state** — `ps aux | grep -iE "mempalace|node.*Better"` should show nothing relevant. Palace should be the 168K skeleton at `~/.mempalace/palace/`.
|
|
2. **Edit `chroma.py`** — file: `/Users/davidglidden/_Dev/mempalace/mempalace/backends/chroma.py`. The `_mempalace_embedding_fn()` function around line 38-43 currently reads:
|
|
```python
|
|
def _mempalace_embedding_fn():
|
|
global _embedding_fn_cache
|
|
if _embedding_fn_cache is None:
|
|
from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
|
|
_embedding_fn_cache = SentenceTransformerEmbeddingFunction(model_name=_MEMPALACE_EMBED_MODEL)
|
|
return _embedding_fn_cache
|
|
```
|
|
Add the `device="mps"` parameter:
|
|
```python
|
|
_embedding_fn_cache = SentenceTransformerEmbeddingFunction(
|
|
model_name=_MEMPALACE_EMBED_MODEL,
|
|
device="mps",
|
|
)
|
|
```
|
|
3. **Sanity test MPS** — quick script to verify bge-m3 loads on Metal and produces 1024-dim output:
|
|
```bash
|
|
~/_Dev/mempalace/.venv/bin/python -c "
|
|
import time, torch
|
|
print('mps available:', torch.backends.mps.is_available())
|
|
from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction
|
|
t0 = time.time()
|
|
ef = SentenceTransformerEmbeddingFunction(model_name='BAAI/bge-m3', device='mps')
|
|
print(f'load: {time.time()-t0:.1f}s')
|
|
t0 = time.time()
|
|
v = ef(['hello', 'bonjour le monde', 'guten tag'])
|
|
print(f'embed 3: {time.time()-t0:.2f}s, dim={len(v[0])}')
|
|
"
|
|
```
|
|
Expected: load ~10s (model cached), embed 3 sentences ~0.1-0.5s, dim=1024.
|
|
4. **Start mine** — same form as before:
|
|
```bash
|
|
nohup ~/_Dev/mempalace/.venv/bin/mempalace mine ~/_Dev/chamber-library/converted_texts --wing chamber_library > /tmp/mine-chamber-library-$(date +%Y-%m-%d).log 2>&1 < /dev/null & disown
|
|
```
|
|
5. **Verify GPU engagement** — within 30 seconds:
|
|
- Activity Monitor → GPU view should show GPU utilization
|
|
- Log shows "Loading weights" then file processing
|
|
- Estimated rate at MPS: ~16-30 files/hr (vs 3.3 files/hr on CPU)
|
|
- Total chamber-library: ~12-24 hours (vs 4-5 days on CPU)
|
|
|
|
## After chamber-library completes
|
|
|
|
Three other wings need re-mining under bge-m3 + MPS, in the steward's preferred order:
|
|
- **Phase 4b** — Obsidian vault (was 112,558 drawers under all-MiniLM-L6-v2). Path: `~/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch`. `--wing obsidian-vault`.
|
|
- **Phase 4c** — Claude transcripts. Path: `~/.claude/projects/-Users-davidglidden`. `--mode convos`. Was 12,688 drawers. (Note: a `Stop` hook auto-fires `mempalace mine` on this directory — saw it spawn an unauthorized parallel mine 2026-04-30 night; if the hook is still active, it'll do this work automatically when sessions end. Otherwise queue manually.)
|
|
- **Phase 4d** — CapableMind thinking docs. Path: `~/_Dev/CapableMind-AI/docs/thinking/David/`. `--wing capablemind-thinking`. Was 6,137 drawers.
|
|
|
|
Likely 1-2 days total for all three under MPS. Could run sequentially overnight on consecutive nights.
|
|
|
|
## Independent thread surfaced — BMF L1 audit
|
|
|
|
Killing BMF (PID 1043) revealed a real L1 pathology that needs deliberate attention:
|
|
|
|
- **1,349 index segments** in `~/.capablemind/data/mindfabric-00/lance/vector_chunks.lance/_indices/` (healthy Lance stores have 1-10)
|
|
- **25,449+ transactions** in the same store
|
|
- 14+ hours of sustained 224% CPU consuming all those segments per query
|
|
- Data store itself is small (139 MB) — the bloat is metadata/segment count, not data volume
|
|
|
|
This is **not** issue #10 (resolved). It's a fresh pathology — likely Lance compaction never ran, or a write-amplification loop in BMF's ingestion path creates a new index segment per drawer/batch.
|
|
|
|
Belongs to: a separate L1 audit session, steward + Seb territory. Lance state is preserved (BMF's killed but data intact); when audit happens, `pkill node` cleanup is already done.
|
|
|
|
Reference forensic notes from prior L1 audit: `~/_Dev/CapableMind-AI/docs/thinking/David/l1-reliability/` (`l1-deep-audit-plan-2026-04-19.md`, `l1-ingestion-forensic-2026-04-19.md`, `l1-recall-quality-baseline-2026-04-19.md`).
|
|
|
|
## Why this transition felt long
|
|
|
|
Steward 2026-05-01: *"I can see how mempalace helps us immensely and want to finish this update transition which I almost regret doing — I didn't think it would take days to do."*
|
|
|
|
Honest accounting:
|
|
- Power failure 2026-04-30 mid-stream cost a full mine + caused the 217 GB HNSW corruption requiring full reset.
|
|
- bge-m3 on CPU is genuinely 5-10x slower than was reasonable to expect for this corpus size. CPU choice was a default-not-a-decision; should have proposed MPS from the start.
|
|
- Multiple parallel processes (auto-Stop-hook transcripts mine + BMF) competed for CPU and confused diagnosis.
|
|
- Each diagnostic round added time without throughput.
|
|
|
|
After the MPS switch the transition should feel proportionate to its value.
|
|
|
|
## Pulling thread for the fresh session
|
|
|
|
**Apply the `device="mps"` patch and restart the chamber-library mine; everything else (palace state, kills, scope decision, model decision) is already done.** Five steps from "open a fresh session" to "mine running on Metal." |