dotfiles/claude/memory/project-bge-m3-full-remine-plan-2026-05-11.md

---
name: bge-m3 full re-mine plan — 2026-05-11
description: The ONE job for the cleared context. Full chamber-library re-mine on
  PR
type: project
originSessionId: a156edbb-0c3d-4285-b3ed-f5a45b8dda66
permalink: claude-memory/project-bge-m3-full-remine-plan-2026-05-11
---

# bge-m3 full re-mine plan

> *"Do things once, correctly, with lasting integrity. Choose what is proportionate, fitting, and durable. Build what you will not need to rebuild."*

This document is the **single load-bearing artifact** for the next session. The cleared context's one job is to execute this plan cleanly. Everything else — Marker history, conversion campaign details, footnote work, library survey — is past. Don't get pulled back into those threads.

## What's already done (before context cleared)

- **PR #442 test validated** — `palace-bge-m3-test/` at `~/.mempalace/palace-bge-m3-test/`, 21,735 drawers from 11 curated multilingual books, mining in 25 min, all 10 search-quality queries passed. Cross-language semantic retrieval (English↔French↔Spanish) verified empirically.
- **Test report posted** to PR #442: https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732
- **PR #442 venv ready**: `~/.local/mempalace-pr442-venv/` with mempalace + sentence-transformers 5.4.1 installed. PR worktree at `~/.local/mempalace-pr442/` on branch `pr442` (head c652c2f3).
- **bge-m3 model cached** at `~/.cache/huggingface/hub/models--BAAI--bge-m3` — no download needed.
- **MCP killed** for clean isolation — no concurrent-writer risk.

## What's now to do

**One job: replace the mxbai 639k palace with a fully-mined bge-m3 palace covering all of chamber-library, before the steward returns from a 48-hour trip.**

### Step 1: Rename palaces for clarity (1 min)

```bash
mv ~/.mempalace/palace ~/.mempalace/palace-mxbai-archive-2026-05-11
mv ~/.mempalace/palace-bge-m3-test ~/.mempalace/palace
```

After this:
- `~/.mempalace/palace/` = bge-m3 (new default, seeded with 21,735 drawers from the test)
- `~/.mempalace/palace-mxbai-archive-2026-05-11/` = the mxbai baseline preserved as queryable archive
- `~/.mempalace/palace.broken-config-2026-05-05/` = leave as-is (legacy fallback)

### Step 2: Verify pre-flight state (2 min)

```bash
# No MCP
pgrep -f mempalace-mcp || echo "✓ MCP stopped"

# PR #442 venv mempalace works
~/.local/mempalace-pr442-venv/bin/mempalace --version          # MemPalace 3.3.3
~/.local/mempalace-pr442-venv/bin/python -c "import sentence_transformers; print(sentence_transformers.__version__)"  # 5.4.1

# Drawer count baseline (the bge-m3 test seed)
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
  'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'
# Expected: 21735
```

### Step 3: Launch the full mine (background, detached)

Mine the rest of chamber-library (the test-palace already has 11 books from the typography/phenomenology subset; mempalace dedup will skip those):

```bash
~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
  ~/_Dev/chamber-library/converted_texts \
  --wing chamber_library
```

Estimated time: ~13-15 hours for ~700k drawers total (~700-800 source markdown files across all `traditions/` and `collections/` subdirs). The dedup will skip the 11 already-mined files automatically.

**Use `run_in_background: true`** so the mine survives session boundaries. Parent will be reparented to launchd on Claude Code exit.

Optional second mine for vault content (Obsidian notes — represents the largest single corpus the old mxbai palace held):

```bash
~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
  "/Users/davidglidden/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch" \
  --wing obsidian_vault
```

(Defer the vault mine until chamber-library wraps cleanly — adds ~5000+ files which is its own substantial mining session. Steward may want to discuss scope when they return.)

### Step 4: Monitor remotely (during steward's 48-hour absence)

Steward will ping via `/remote-control` periodically. When they do:

```bash
# Mining still running?
ps -ef | grep "mempalace.*mine" | grep -v grep

# Drawer count + delta from last check
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
  'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'

# Last few mine log entries
tail -20 ~/.mempalace/hook_state/mine.log 2>/dev/null

# Disk usage
du -sh ~/.mempalace/palace
```

Report: % complete (rough), any errors, ETA.

## Risk + recovery profile (read carefully)

**The risk:** PR #442's branch does NOT yet contain 3.3.5's concurrent-writer lock fix (#1162) or HNSW quarantine helpers (#1339, #1342). A long mine carries non-zero chance of HNSW corruption — same class of failure that broke the earlier 933k bge-m3 attempt.

**Mitigation:**
- MCP stays killed throughout. Single writer = deterministic.
- No other Claude Code session active (steward confirmed).
- Auto-save hooks won't fire (MCP is the trigger; MCP is dead).

**Recovery if HNSW corrupts mid-mine:**
- The sqlite layer (chroma.sqlite3) is durable; survives even when HNSW segments rot. Per the verification in #1401, 136k drawers can be recovered from a corrupted-HNSW palace via SQL replay.
- Recovery tool ships in 3.3.5: `mempalace repair --mode from-sqlite --archive-existing`. 74 min wall-clock per verified test.
- If corruption happens, DON'T wipe the palace. Wait for 3.3.5 to ship; run repair; back in business.

**Recovery if mine fails mid-run:**
- mempalace mine is idempotent on re-mine (deterministic IDs; dedup skips already-filed files).
- Just re-run the same command. It picks up where it left off.

## Transition path when 3.3.5 + #442 lands on main

When the upstream maintainers ship 3.3.5 (release branch already prepped at PR #1434) and merge #442 onto develop+main:

```bash
# Upgrade the system mempalace install
cd ~/_Dev/mempalace
git pull
pip install -e .

# Palace is unchanged — collection metadata tells mempalace to use bge-m3
# MCP server (if started) connects to the default palace at ~/.mempalace/palace
# Auto-save hooks file to the bge-m3 palace
# Everything just works

# Retire the PR #442 venv (no longer needed)
git -C ~/_Dev/mempalace worktree remove ~/.local/mempalace-pr442 --force
rm -rf ~/.local/mempalace-pr442-venv
```

Vector data is durable across mempalace versions. No re-mine needed at the transition.

## What NOT to do (deferred)

- **Don't touch ARC SCSS / AldineXXI spec work** — that's a separate workstream in a different session.
- **Don't engage Marker** — Marker is deprecated for our needs; Docling is the conversion tool. (The conversion campaign is done; all PDFs are converted.)
- **Don't survey the source library** — the cleanup of `~/Documents/The Library [ePub_AWZ3]/` is post-mining work. Steward will handle when they return.
- **Don't write to or modify chamber-library content** — only read for mining.
- **Don't restart MCP** — leave it dead until steward returns and decides.
- **Don't `kill` the mine process** unless explicitly asked. If mine hangs or errors, surface to steward via remote-control; don't unilaterally kill.

## Files / paths quick reference

| Thing | Path |
|---|---|
| New default palace (bge-m3, becomes the production palace) | `~/.mempalace/palace/` |
| Old mxbai palace (preserved as archive) | `~/.mempalace/palace-mxbai-archive-2026-05-11/` |
| PR #442 mempalace binary | `~/.local/mempalace-pr442-venv/bin/mempalace` |
| PR #442 worktree | `~/.local/mempalace-pr442/` (branch: `pr442`) |
| Source content to mine | `~/_Dev/chamber-library/converted_texts/` |
| PR comment posted | https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732 |

## Pulling thread (singular)

**Re-mine the chamber-library cleanly on bge-m3, survive the 48 hours without writer collisions or HNSW corruption, report status when steward pings.**

Everything else waits.