dotfiles/claude/memory/project-bge-m3-full-remine-plan-2026-05-11.md

name, description, type, originSessionId, permalink

name	description	type	originSessionId	permalink
bge-m3 full re-mine plan — 2026-05-11	The ONE job for the cleared context. Full chamber-library re-mine on PR	project	a156edbb-0c3d-4285-b3ed-f5a45b8dda66	claude-memory/project-bge-m3-full-remine-plan-2026-05-11

bge-m3 full re-mine plan

"Do things once, correctly, with lasting integrity. Choose what is proportionate, fitting, and durable. Build what you will not need to rebuild."

This document is the single load-bearing artifact for the next session. The cleared context's one job is to execute this plan cleanly. Everything else — Marker history, conversion campaign details, footnote work, library survey — is past. Don't get pulled back into those threads.

What's already done (before context cleared)

• PR #442 test validated — palace-bge-m3-test/ at ~/.mempalace/palace-bge-m3-test/, 21,735 drawers from 11 curated multilingual books, mining in 25 min, all 10 search-quality queries passed. Cross-language semantic retrieval (English↔French↔Spanish) verified empirically.
• Test report posted to PR #442: https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732
• PR #442 venv ready: ~/.local/mempalace-pr442-venv/ with mempalace + sentence-transformers 5.4.1 installed. PR worktree at ~/.local/mempalace-pr442/ on branch pr442 (head c652c2f3).
• bge-m3 model cached at ~/.cache/huggingface/hub/models--BAAI--bge-m3 — no download needed.
• MCP killed for clean isolation — no concurrent-writer risk.

What's now to do

One job: replace the mxbai 639k palace with a fully-mined bge-m3 palace covering all of chamber-library, before the steward returns from a 48-hour trip.

Step 1: Rename palaces for clarity (1 min)

mv ~/.mempalace/palace ~/.mempalace/palace-mxbai-archive-2026-05-11
mv ~/.mempalace/palace-bge-m3-test ~/.mempalace/palace

After this:

• ~/.mempalace/palace/ = bge-m3 (new default, seeded with 21,735 drawers from the test)
• ~/.mempalace/palace-mxbai-archive-2026-05-11/ = the mxbai baseline preserved as queryable archive
• ~/.mempalace/palace.broken-config-2026-05-05/ = leave as-is (legacy fallback)

Step 2: Verify pre-flight state (2 min)

# No MCP
pgrep -f mempalace-mcp || echo "✓ MCP stopped"

# PR #442 venv mempalace works
~/.local/mempalace-pr442-venv/bin/mempalace --version          # MemPalace 3.3.3
~/.local/mempalace-pr442-venv/bin/python -c "import sentence_transformers; print(sentence_transformers.__version__)"  # 5.4.1

# Drawer count baseline (the bge-m3 test seed)
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
  'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'
# Expected: 21735

Step 3: Launch the full mine (background, detached)

Mine the rest of chamber-library (the test-palace already has 11 books from the typography/phenomenology subset; mempalace dedup will skip those):

~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
  ~/_Dev/chamber-library/converted_texts \
  --wing chamber_library

Estimated time: ~13-15 hours for ~700k drawers total (~700-800 source markdown files across all traditions/ and collections/ subdirs). The dedup will skip the 11 already-mined files automatically.

Use run_in_background: true so the mine survives session boundaries. Parent will be reparented to launchd on Claude Code exit.

Optional second mine for vault content (Obsidian notes — represents the largest single corpus the old mxbai palace held):

~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
  "/Users/davidglidden/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch" \
  --wing obsidian_vault

(Defer the vault mine until chamber-library wraps cleanly — adds ~5000+ files which is its own substantial mining session. Steward may want to discuss scope when they return.)

Step 4: Monitor remotely (during steward's 48-hour absence)

Steward will ping via /remote-control periodically. When they do:

# Mining still running?
ps -ef | grep "mempalace.*mine" | grep -v grep

# Drawer count + delta from last check
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
  'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'

# Last few mine log entries
tail -20 ~/.mempalace/hook_state/mine.log 2>/dev/null

# Disk usage
du -sh ~/.mempalace/palace

Report: % complete (rough), any errors, ETA.

Risk + recovery profile (read carefully)

The risk: PR #442's branch does NOT yet contain 3.3.5's concurrent-writer lock fix (#1162) or HNSW quarantine helpers (#1339, #1342). A long mine carries non-zero chance of HNSW corruption — same class of failure that broke the earlier 933k bge-m3 attempt.

Mitigation:

• MCP stays killed throughout. Single writer = deterministic.
• No other Claude Code session active (steward confirmed).
• Auto-save hooks won't fire (MCP is the trigger; MCP is dead).

Recovery if HNSW corrupts mid-mine:

• The sqlite layer (chroma.sqlite3) is durable; survives even when HNSW segments rot. Per the verification in #1401, 136k drawers can be recovered from a corrupted-HNSW palace via SQL replay.
• Recovery tool ships in 3.3.5: mempalace repair --mode from-sqlite --archive-existing. 74 min wall-clock per verified test.
• If corruption happens, DON'T wipe the palace. Wait for 3.3.5 to ship; run repair; back in business.

Recovery if mine fails mid-run:

• mempalace mine is idempotent on re-mine (deterministic IDs; dedup skips already-filed files).
• Just re-run the same command. It picks up where it left off.

Transition path when 3.3.5 + #442 lands on main

When the upstream maintainers ship 3.3.5 (release branch already prepped at PR #1434) and merge #442 onto develop+main:

# Upgrade the system mempalace install
cd ~/_Dev/mempalace
git pull
pip install -e .

# Palace is unchanged — collection metadata tells mempalace to use bge-m3
# MCP server (if started) connects to the default palace at ~/.mempalace/palace
# Auto-save hooks file to the bge-m3 palace
# Everything just works

# Retire the PR #442 venv (no longer needed)
git -C ~/_Dev/mempalace worktree remove ~/.local/mempalace-pr442 --force
rm -rf ~/.local/mempalace-pr442-venv

Vector data is durable across mempalace versions. No re-mine needed at the transition.

What NOT to do (deferred)

• Don't touch ARC SCSS / AldineXXI spec work — that's a separate workstream in a different session.
• Don't engage Marker — Marker is deprecated for our needs; Docling is the conversion tool. (The conversion campaign is done; all PDFs are converted.)
• Don't survey the source library — the cleanup of ~/Documents/The Library [ePub_AWZ3]/ is post-mining work. Steward will handle when they return.
• Don't write to or modify chamber-library content — only read for mining.
• Don't restart MCP — leave it dead until steward returns and decides.
• Don't kill the mine process unless explicitly asked. If mine hangs or errors, surface to steward via remote-control; don't unilaterally kill.

Files / paths quick reference

Thing	Path
New default palace (bge-m3, becomes the production palace)	~/.mempalace/palace/
Old mxbai palace (preserved as archive)	~/.mempalace/palace-mxbai-archive-2026-05-11/
PR #442 mempalace binary	~/.local/mempalace-pr442-venv/bin/mempalace
PR #442 worktree	~/.local/mempalace-pr442/ (branch: pr442)
Source content to mine	~/_Dev/chamber-library/converted_texts/
PR comment posted	https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732

Pulling thread (singular)

Re-mine the chamber-library cleanly on bge-m3, survive the 48 hours without writer collisions or HNSW corruption, report status when steward pings.

Everything else waits.