David F GliddenandClaude Opus 4.8
3f9a89b00c
Basic Memory v0.21.6 first sync over the live memory dir (steward-authorized live-dir trial, Option A 2026-06-06): adds permalink: to frontmatter, refolds long YAML description lines, strips final newlines. Bodies untouched — verified via full diff classification. From this commit forward, any diff in claude/memory shows only what Basic Memory or the session writes. Trial design: MemPalace untouched as incumbent; git status check on this dir at every wrap; end-of-day evaluation (recall quality, sync robustness, rebuild-from-files, malformed-file behavior). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
50 lines
4.4 KiB
Markdown
50 lines
4.4 KiB
Markdown
---
|
|
name: MemPalace chamber-library re-mine — incremental strategy, never another multi-day
|
|
loss
|
|
description: After 3 days lost to bge-m3+MPS chamber-library mine that produced unusable
|
|
storage (chromadb format corruption + global embedding patch conflict), steward
|
|
set the rule. Future library mining is incremental, prioritized by criticality,
|
|
never a single multi-day pass.
|
|
type: project
|
|
originSessionId: 45a40b1e-09cd-42bb-a41a-7298f78a8cf0
|
|
permalink: claude-memory/project-mempalace-library-incremental-remine-strategy
|
|
---
|
|
|
|
## Context
|
|
|
|
The 2026-05-01 → 2026-05-03 chamber-library mine on bge-m3+MPS produced 933,576 drawers in storage but unreadable by chromadb 1.5.8 (format error: "Palace is NOT readable by chromadb 1.5.8"). Three days of CPU/MPS lost. Palace preserved at `~/.mempalace/palace.bge-m3-broken-2026-05-04/` for later forensic recovery.
|
|
|
|
Compounding architectural issue: the local patch in `mempalace/backends/chroma.py` (`_mempalace_embedding_fn`) forced bge-m3 (1024d) for ALL collections globally. This created a hard conflict with the pre-bge-m3 palace (mxbai 384d). Reverted 2026-05-04 to restore baseline functionality on the 131,418 pre-bge-m3 drawers (Obsidian vault, Claude transcripts, etc.). Patch saved at `~/_Dev/mempalace/.local-data/bge-m3-patch-2026-04-30.diff` for reference.
|
|
|
|
## Steward's rule (verbatim)
|
|
|
|
> *"go with path A. then we make a choice on how to remine the library incrementally beginning with the most critical texts. I'm not losing days to this again."*
|
|
|
|
— David Glidden, 2026-05-04
|
|
|
|
## The rule, expanded
|
|
|
|
1. **No single multi-day mine of chamber-library ever again.** A failure that's only visible after days of compute is unacceptable. Mines are incremental and validated end-to-end (storage AND search) before committing more compute.
|
|
|
|
2. **Prioritize by criticality.** The library has many texts; only some are load-bearing for active ARC work. Start with what's actually being used (recently cited / actively being read / referenced in current spec work).
|
|
|
|
3. **Validate after each tranche.** Storage write + search readback must both succeed before the next tranche starts. The bge-m3 mine's failure mode — storage succeeds, search corrupts silently — must be impossible going forward.
|
|
|
|
4. **The architectural fix for the global-patch conflict** is per-collection embedding configuration: bge-m3 for `chamber_library` (multilingual content), default mxbai for everything else. The current saved patch forces one global model and is not a viable long-term answer.
|
|
|
|
## Open questions for the next library work session
|
|
|
|
1. **Triage**: which library texts are actually load-bearing right now? (probably: Bringhurst, Manutius the Younger, Berger, Athenaeus, Vico, Harrison, the Observer Problem sources, anything cited in active essays — but the steward should confirm)
|
|
2. **Forensic on the broken palace**: is the data recoverable, or do we re-mine source files? Cheapest first move: try `mempalace migrate` on `palace.bge-m3-broken-2026-05-04/` (the format error is exactly what `migrate` exists for; it might just work). Status 2026-05-04: dry-run on big palace started but was killed when scope shifted to baseline restore.
|
|
3. **Embedding strategy**: continue with bge-m3 for multilingual quality, OR accept default mxbai for English-only criticality? bge-m3 quality non-negotiable per 2026-05-01 decision (chamber-library has critical French + some German texts), so the architectural fix (per-collection embedding) is the right path.
|
|
4. **Per-collection embedding patch**: refactor the saved patch in `.local-data/bge-m3-patch-2026-04-30.diff` to apply only when collection name matches `chamber_library*` (or via explicit allow-list). Should be ~10-line change. Test against a 100-file mine before any larger commitment.
|
|
|
|
## Anti-patterns to refuse
|
|
|
|
- "Just re-run the mine, it'll probably work this time" — three days lost to that posture.
|
|
- "Skip the dry-run / verification step, we already know it works" — the bge-m3 mine wrote 933k drawers that look ready by `mempalace_status` and fail on first search. Storage success ≠ recall success.
|
|
- "It's only multilingual, English content might still work without bge-m3" — quality assessment was already made: chamber-library has critical French + German content. Don't relitigate.
|
|
|
|
## Status 2026-05-04
|
|
|
|
Baseline restored. Library re-mine strategy parked until steward bandwidth + decision on triage criteria. Forensic on broken big palace is a candidate first move (cheap if it works). |