David/dotfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2f6964cf0e99f1f0c2bbc9004828c8d496e6918d
dotfiles/claude/memory/project-bge-m3-full-remine-plan-2026-05-11.md
T
David F GliddenandClaude Opus 5 ef6fa94dcd [FIX] Stamp 16 superseded trackers; harvest the rule one of them carried 
The record is updated where information ARRIVES and never where a reader LOOKS.
Three instances found the same day, each with the correct information already
written down elsewhere:

  REVIEWED-81 (AUTHORIZED 07-28)      -> PENDING-81 still reads open
  the ARC open-work register, which   -> those 6 trackers still read live
    says in its own description that
    it supersedes them, with a
    per-file verified verdict
  register entry A2: "the content     -> register entry B3, one page below:
    half (B3) is now unblocked +         "[OPEN - gated on A2]"
    started" (06-17)

The third is the sharpest: both halves sit in the SAME document, so no tooling
gap explains it. ~/CLAUDE.md already rules this under Memory Discipline
("mark the superseded record as superseded"); the rule is being applied to facts
and not to trackers, PENDING items or register entries, which are also facts
about state.

Executed under steward authorization, arrears only:
- 16 trackers stamped `superseded_by:` + `superseded_on:` AND a visible in-body
  banner. The banner is the point — metadata alone repeats the defect, since the
  finding is about what a reader sees. 9 obsoleted by the 2026-07-07 MemPalace
  wind-down; 7 already carrying the register's own verified `DONE-retire`.
- B3's label corrected, with the two-month error left visible rather than
  silently amended.
- Nothing moved, renamed or deleted: 381 pointers resolve, 0 dead, 0 mis-authored.

HARVESTED BEFORE STAMPING, and this is why the harvest came first:
`project-mempalace-library-incremental-remine-strategy` carried a steward-verbatim
rule that is instrument-independent — never run a single multi-day index; order by
criticality; gate each tranche on READBACK, not on write-success (933,576 drawers
once stored successfully and could not be read back at all). It lived only inside
a MemPalace tracker, so retiring the instrument would have retired the rule with
it. ~/CLAUDE.md names that exact hazard: "state the obligation first and the
instrument second, or the next retired tool takes a rule down with it." Now
`feedback-bulk-indexing-runs-incrementally-with-readback.md`, and it applies to
studium-engine corpus work.

Held back deliberately: 2 correctly-parked files with named dependencies, 2 named
nowhere (returned to the steward), and 39 trackers unread — recorded as a gap,
not as a finding.

Measurement note: every earlier estimate of tracker staleness was wrong, because
both mtime and git-last-commit were reset across 283 files by the 2026-06-06
normalization sweep. The honest figure excludes it: median 90 days.

Filed as PENDING-142 ADDENDUM 2 per steward direction (fold, do not file anew).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Y6t6qx7cpaCu5xGdD36u4
2026-08-17 15:23:26 +02:00

8.3 KiB
Raw Blame History

name, description, type, originSessionId, permalink, superseded_by, superseded_on
name description type originSessionId permalink superseded_by superseded_on
bge-m3 full re-mine plan — 2026-05-11 The ONE job for the cleared context. Full chamber-library re-mine on PR project a156edbb-0c3d-4285-b3ed-f5a45b8dda66 claude-memory/project-bge-m3-full-remine-plan-2026-05-11 project-mempalace-winddown.md 2026-08-17

SUPERSEDED 2026-08-17. An execution plan for a palace that no longer exists. The DISCIPLINE it was written under survives: see feedback-bulk-indexing-runs-incrementally-with-readback.md.

Current record: project-mempalace-winddown.md. Kept for detail and provenance — do not read the status below as live.

bge-m3 full re-mine plan

"Do things once, correctly, with lasting integrity. Choose what is proportionate, fitting, and durable. Build what you will not need to rebuild."

This document is the single load-bearing artifact for the next session. The cleared context's one job is to execute this plan cleanly. Everything else — Marker history, conversion campaign details, footnote work, library survey — is past. Don't get pulled back into those threads.

What's already done (before context cleared)

  • PR #442 test validated — palace-bge-m3-test/ at ~/.mempalace/palace-bge-m3-test/, 21,735 drawers from 11 curated multilingual books, mining in 25 min, all 10 search-quality queries passed. Cross-language semantic retrieval (English↔French↔Spanish) verified empirically.
  • Test report posted to PR #442: https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732
  • PR #442 venv ready: ~/.local/mempalace-pr442-venv/ with mempalace + sentence-transformers 5.4.1 installed. PR worktree at ~/.local/mempalace-pr442/ on branch pr442 (head c652c2f3).
  • bge-m3 model cached at ~/.cache/huggingface/hub/models--BAAI--bge-m3 — no download needed.
  • MCP killed for clean isolation — no concurrent-writer risk.

What's now to do

One job: replace the mxbai 639k palace with a fully-mined bge-m3 palace covering all of chamber-library, before the steward returns from a 48-hour trip.

Step 1: Rename palaces for clarity (1 min)

mv ~/.mempalace/palace ~/.mempalace/palace-mxbai-archive-2026-05-11
mv ~/.mempalace/palace-bge-m3-test ~/.mempalace/palace

After this:

  • ~/.mempalace/palace/ = bge-m3 (new default, seeded with 21,735 drawers from the test)
  • ~/.mempalace/palace-mxbai-archive-2026-05-11/ = the mxbai baseline preserved as queryable archive
  • ~/.mempalace/palace.broken-config-2026-05-05/ = leave as-is (legacy fallback)

Step 2: Verify pre-flight state (2 min)

# No MCP
pgrep -f mempalace-mcp || echo "✓ MCP stopped"

# PR #442 venv mempalace works
~/.local/mempalace-pr442-venv/bin/mempalace --version          # MemPalace 3.3.3
~/.local/mempalace-pr442-venv/bin/python -c "import sentence_transformers; print(sentence_transformers.__version__)"  # 5.4.1

# Drawer count baseline (the bge-m3 test seed)
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
  'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'
# Expected: 21735

Step 3: Launch the full mine (background, detached)

Mine the rest of chamber-library (the test-palace already has 11 books from the typography/phenomenology subset; mempalace dedup will skip those):

~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
  ~/_Dev/chamber-library/converted_texts \
  --wing chamber_library

Estimated time: ~13-15 hours for ~700k drawers total (~700-800 source markdown files across all traditions/ and collections/ subdirs). The dedup will skip the 11 already-mined files automatically.

Use run_in_background: true so the mine survives session boundaries. Parent will be reparented to launchd on Claude Code exit.

Optional second mine for vault content (Obsidian notes — represents the largest single corpus the old mxbai palace held):

~/.local/mempalace-pr442-venv/bin/mempalace --palace ~/.mempalace/palace mine \
  "/Users/davidglidden/Library/Mobile Documents/iCloud~md~obsidian/Documents/David, root-and-branch" \
  --wing obsidian_vault

(Defer the vault mine until chamber-library wraps cleanly — adds ~5000+ files which is its own substantial mining session. Steward may want to discuss scope when they return.)

Step 4: Monitor remotely (during steward's 48-hour absence)

Steward will ping via /remote-control periodically. When they do:

# Mining still running?
ps -ef | grep "mempalace.*mine" | grep -v grep

# Drawer count + delta from last check
sqlite3 ~/.mempalace/palace/chroma.sqlite3 \
  'SELECT COUNT(*) FROM embeddings e JOIN segments s ON e.segment_id=s.id JOIN collections c ON s.collection=c.id WHERE c.name="mempalace_drawers";'

# Last few mine log entries
tail -20 ~/.mempalace/hook_state/mine.log 2>/dev/null

# Disk usage
du -sh ~/.mempalace/palace

Report: % complete (rough), any errors, ETA.

Risk + recovery profile (read carefully)

The risk: PR #442's branch does NOT yet contain 3.3.5's concurrent-writer lock fix (#1162) or HNSW quarantine helpers (#1339, #1342). A long mine carries non-zero chance of HNSW corruption — same class of failure that broke the earlier 933k bge-m3 attempt.

Mitigation:

  • MCP stays killed throughout. Single writer = deterministic.
  • No other Claude Code session active (steward confirmed).
  • Auto-save hooks won't fire (MCP is the trigger; MCP is dead).

Recovery if HNSW corrupts mid-mine:

  • The sqlite layer (chroma.sqlite3) is durable; survives even when HNSW segments rot. Per the verification in #1401, 136k drawers can be recovered from a corrupted-HNSW palace via SQL replay.
  • Recovery tool ships in 3.3.5: mempalace repair --mode from-sqlite --archive-existing. 74 min wall-clock per verified test.
  • If corruption happens, DON'T wipe the palace. Wait for 3.3.5 to ship; run repair; back in business.

Recovery if mine fails mid-run:

  • mempalace mine is idempotent on re-mine (deterministic IDs; dedup skips already-filed files).
  • Just re-run the same command. It picks up where it left off.

Transition path when 3.3.5 + #442 lands on main

When the upstream maintainers ship 3.3.5 (release branch already prepped at PR #1434) and merge #442 onto develop+main:

# Upgrade the system mempalace install
cd ~/_Dev/mempalace
git pull
pip install -e .

# Palace is unchanged — collection metadata tells mempalace to use bge-m3
# MCP server (if started) connects to the default palace at ~/.mempalace/palace
# Auto-save hooks file to the bge-m3 palace
# Everything just works

# Retire the PR #442 venv (no longer needed)
git -C ~/_Dev/mempalace worktree remove ~/.local/mempalace-pr442 --force
rm -rf ~/.local/mempalace-pr442-venv

Vector data is durable across mempalace versions. No re-mine needed at the transition.

What NOT to do (deferred)

  • Don't touch ARC SCSS / AldineXXI spec work — that's a separate workstream in a different session.
  • Don't engage Marker — Marker is deprecated for our needs; Docling is the conversion tool. (The conversion campaign is done; all PDFs are converted.)
  • Don't survey the source library — the cleanup of ~/Documents/The Library [ePub_AWZ3]/ is post-mining work. Steward will handle when they return.
  • Don't write to or modify chamber-library content — only read for mining.
  • Don't restart MCP — leave it dead until steward returns and decides.
  • Don't kill the mine process unless explicitly asked. If mine hangs or errors, surface to steward via remote-control; don't unilaterally kill.

Files / paths quick reference

Thing Path
New default palace (bge-m3, becomes the production palace) ~/.mempalace/palace/
Old mxbai palace (preserved as archive) ~/.mempalace/palace-mxbai-archive-2026-05-11/
PR #442 mempalace binary ~/.local/mempalace-pr442-venv/bin/mempalace
PR #442 worktree ~/.local/mempalace-pr442/ (branch: pr442)
Source content to mine ~/_Dev/chamber-library/converted_texts/
PR comment posted https://github.com/MemPalace/mempalace/pull/442#issuecomment-4418237732

Pulling thread (singular)

Re-mine the chamber-library cleanly on bge-m3, survive the 48 hours without writer collisions or HNSW corruption, report status when steward pings.

Everything else waits.

Reference in New Issue View Git Blame Copy Permalink