David/dotfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ef6fa94dcdd0c05c0cce1e3fbeacf288f37ba497
dotfiles/claude/memory/project-bge-m3-full-remine-plan-2026-05-11.md
T
David F GliddenandClaude Opus 5 ef6fa94dcd [FIX] Stamp 16 superseded trackers; harvest the rule one of them carried 
The record is updated where information ARRIVES and never where a reader LOOKS.
Three instances found the same day, each with the correct information already
written down elsewhere:

  REVIEWED-81 (AUTHORIZED 07-28)      -> PENDING-81 still reads open
  the ARC open-work register, which   -> those 6 trackers still read live
    says in its own description that
    it supersedes them, with a
    per-file verified verdict
  register entry A2: "the content     -> register entry B3, one page below:
    half (B3) is now unblocked +         "[OPEN - gated on A2]"
    started" (06-17)

The third is the sharpest: both halves sit in the SAME document, so no tooling
gap explains it. ~/CLAUDE.md already rules this under Memory Discipline
("mark the superseded record as superseded"); the rule is being applied to facts
and not to trackers, PENDING items or register entries, which are also facts
about state.

Executed under steward authorization, arrears only:
- 16 trackers stamped `superseded_by:` + `superseded_on:` AND a visible in-body
  banner. The banner is the point — metadata alone repeats the defect, since the
  finding is about what a reader sees. 9 obsoleted by the 2026-07-07 MemPalace
  wind-down; 7 already carrying the register's own verified `DONE-retire`.
- B3's label corrected, with the two-month error left visible rather than
  silently amended.
- Nothing moved, renamed or deleted: 381 pointers resolve, 0 dead, 0 mis-authored.

HARVESTED BEFORE STAMPING, and this is why the harvest came first:
`project-mempalace-library-incremental-remine-strategy` carried a steward-verbatim
rule that is instrument-independent — never run a single multi-day index; order by
criticality; gate each tranche on READBACK, not on write-success (933,576 drawers
once stored successfully and could not be read back at all). It lived only inside
a MemPalace tracker, so retiring the instrument would have retired the rule with
it. ~/CLAUDE.md names that exact hazard: "state the obligation first and the
instrument second, or the next retired tool takes a rule down with it." Now
`feedback-bulk-indexing-runs-incrementally-with-readback.md`, and it applies to
studium-engine corpus work.

Held back deliberately: 2 correctly-parked files with named dependencies, 2 named
nowhere (returned to the steward), and 39 trackers unread — recorded as a gap,
not as a finding.

Measurement note: every earlier estimate of tracker staleness was wrong, because
both mtime and git-last-commit were reset across 283 files by the 2026-06-06
normalization sweep. The honest figure excludes it: median 90 days.

Filed as PENDING-142 ADDENDUM 2 per steward direction (fold, do not file anew).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Y6t6qx7cpaCu5xGdD36u4
2026-08-17 15:23:26 +02:00

172 lines
8.3 KiB
Markdown
Raw Blame History

---
name: bge-m3 full re-mine plan — 2026-05-11
description: The ONE job for the cleared context. Full chamber-library re-mine on
PR
type: project
originSessionId: a156edbb-0c3d-4285-b3ed-f5a45b8dda66
permalink: claude-memory/project-bge-m3-full-remine-plan-2026-05-11
superseded_by: project-mempalace-winddown.md
superseded_on: 2026-08-17
---
> **SUPERSEDED 2026-08-17.** An execution plan for a palace that no longer exists. The DISCIPLINE it was written under survives: see feedback-bulk-indexing-runs-incrementally-with-readback.md.
>
> Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the
> status below as live.
# bge-m3 full re-mine plan
> *"Do things once, correctly, with lasting integrity. Choose what is proportionate, fitting, and durable. Build what you will not need to rebuild."*
This document is the **single load-bearing artifact** for the next session. The cleared context's one job is to execute this plan cleanly. Everything else — Marker history, conversion campaign details, footnote work, library survey — is past. Don't get pulled back into those threads.
## What's already done (before context cleared)
- **PR #442 test validated** — `palace-bge-m3-test/` at `~/.mempalace/palace-bge-m3-test/`, 21,735 drawers from 11 curated multilingual books, mining in 25 min, all 10 search-quality queries passed. Cross-language semantic retrieval (English↔French↔Spanish) verified empirically.
- **Test report posted** to PR #442: https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732
- **PR #442 venv ready**: `~/.local/mempalace-pr442-venv/` with mempalace + sentence-transformers 5.4.1 installed. PR worktree at `~/.local/mempalace-pr442/` on branch `pr442` (head c652c2f3).
- **bge-m3 model cached** at `~/.cache/huggingface/hub/models--BAAI--bge-m3` — no download needed.
- **MCP killed** for clean isolation — no concurrent-writer risk.
## What's now to do
**One job: replace the mxbai 639k palace with a fully-mined bge-m3 palace covering all of chamber-library, before the steward returns from a 48-hour trip.**
### Step 1: Rename palaces for clarity (1 min)
```bash
mv ~/.mempalace/palace ~/.mempalace/palace-mxbai-archive-2026-05-11
mv ~/.mempalace/palace-bge-m3-test ~/.mempalace/palace
```
After this:
- `~/.mempalace/palace/` = bge-m3 (new default, seeded with 21,735 drawers from the test)
- `~/.mempalace/palace-mxbai-archive-2026-05-11/` = the mxbai baseline preserved as queryable archive
- `~/.mempalace/palace.broken-config-2026-05-05/` = leave as-is (legacy fallback)
### Step 2: Verify pre-flight state (2 min)
```bash
# No MCP
pgrep -f mempalace-mcp || echo "✓ MCP stopped"
# PR #442 venv mempalace works
~/.local/mempalace-pr442-venv/bin/mempalace --version # MemPalace 3.3.3
~/.local/mempalace-pr442-venv/bin/python -c "import sentence_transformers; print(sentence_transformers.__version__)" # 5.4.1
# Drawer count baseline (the bge-m3 test seed)
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'
# Expected: 21735
```
### Step 3: Launch the full mine (background, detached)
Mine the rest of chamber-library (the test-palace already has 11 books from the typography/phenomenology subset; mempalace dedup will skip those):
```bash
~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
~/_Dev/chamber-library/converted_texts \
--wing chamber_library
```
Estimated time: ~13-15 hours for ~700k drawers total (~700-800 source markdown files across all `traditions/` and `collections/` subdirs). The dedup will skip the 11 already-mined files automatically.
**Use `run_in_background: true`** so the mine survives session boundaries. Parent will be reparented to launchd on Claude Code exit.
Optional second mine for vault content (Obsidian notes — represents the largest single corpus the old mxbai palace held):
```bash
~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
"/Users/davidglidden/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch" \
--wing obsidian_vault
```
(Defer the vault mine until chamber-library wraps cleanly — adds ~5000+ files which is its own substantial mining session. Steward may want to discuss scope when they return.)
### Step 4: Monitor remotely (during steward's 48-hour absence)
Steward will ping via `/remote-control` periodically. When they do:
```bash
# Mining still running?
ps -ef | grep "mempalace.*mine" | grep -v grep
# Drawer count + delta from last check
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'
# Last few mine log entries
tail -20 ~/.mempalace/hook_state/mine.log 2>/dev/null
# Disk usage
du -sh ~/.mempalace/palace
```
Report: % complete (rough), any errors, ETA.
## Risk + recovery profile (read carefully)
**The risk:** PR #442's branch does NOT yet contain 3.3.5's concurrent-writer lock fix (#1162) or HNSW quarantine helpers (#1339, #1342). A long mine carries non-zero chance of HNSW corruption — same class of failure that broke the earlier 933k bge-m3 attempt.
**Mitigation:**
- MCP stays killed throughout. Single writer = deterministic.
- No other Claude Code session active (steward confirmed).
- Auto-save hooks won't fire (MCP is the trigger; MCP is dead).
**Recovery if HNSW corrupts mid-mine:**
- The sqlite layer (chroma.sqlite3) is durable; survives even when HNSW segments rot. Per the verification in #1401, 136k drawers can be recovered from a corrupted-HNSW palace via SQL replay.
- Recovery tool ships in 3.3.5: `mempalace repair --mode from-sqlite --archive-existing`. 74 min wall-clock per verified test.
- If corruption happens, DON'T wipe the palace. Wait for 3.3.5 to ship; run repair; back in business.
**Recovery if mine fails mid-run:**
- mempalace mine is idempotent on re-mine (deterministic IDs; dedup skips already-filed files).
- Just re-run the same command. It picks up where it left off.
## Transition path when 3.3.5 + #442 lands on main
When the upstream maintainers ship 3.3.5 (release branch already prepped at PR #1434) and merge #442 onto develop+main:
```bash
# Upgrade the system mempalace install
cd ~/_Dev/mempalace
git pull
pip install -e .
# Palace is unchanged — collection metadata tells mempalace to use bge-m3
# MCP server (if started) connects to the default palace at ~/.mempalace/palace
# Auto-save hooks file to the bge-m3 palace
# Everything just works
# Retire the PR #442 venv (no longer needed)
git -C ~/_Dev/mempalace worktree remove ~/.local/mempalace-pr442 --force
rm -rf ~/.local/mempalace-pr442-venv
```
Vector data is durable across mempalace versions. No re-mine needed at the transition.
## What NOT to do (deferred)
- **Don't touch ARC SCSS / AldineXXI spec work** — that's a separate workstream in a different session.
- **Don't engage Marker** — Marker is deprecated for our needs; Docling is the conversion tool. (The conversion campaign is done; all PDFs are converted.)
- **Don't survey the source library** — the cleanup of `~/Documents/The Library [ePub_AWZ3]/` is post-mining work. Steward will handle when they return.
- **Don't write to or modify chamber-library content** — only read for mining.
- **Don't restart MCP** — leave it dead until steward returns and decides.
- **Don't `kill` the mine process** unless explicitly asked. If mine hangs or errors, surface to steward via remote-control; don't unilaterally kill.
## Files / paths quick reference
| Thing | Path |
|---|---|
| New default palace (bge-m3, becomes the production palace) | `~/.mempalace/palace/` |
| Old mxbai palace (preserved as archive) | `~/.mempalace/palace-mxbai-archive-2026-05-11/` |
| PR #442 mempalace binary | `~/.local/mempalace-pr442-venv/bin/mempalace` |
| PR #442 worktree | `~/.local/mempalace-pr442/` (branch: `pr442`) |
| Source content to mine | `~/_Dev/chamber-library/converted_texts/` |
| PR comment posted | https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732 |
## Pulling thread (singular)
**Re-mine the chamber-library cleanly on bge-m3, survive the 48 hours without writer collisions or HNSW corruption, report status when steward pings.**
Everything else waits.
Reference in New Issue View Git Blame Copy Permalink