---
name: MemPalace chamber-library re-mine — incremental strategy, never another multi-day
  loss
description: After 3 days lost to bge-m3+MPS chamber-library mine that produced unusable
  storage (chromadb format corruption + global embedding patch conflict), steward
  set the rule. Future library mining is incremental, prioritized by criticality,
  never a single multi-day pass.
type: project
originSessionId: 45a40b1e-09cd-42bb-a41a-7298f78a8cf0
permalink: claude-memory/project-mempalace-library-incremental-remine-strategy
superseded_by: project-mempalace-winddown.md
superseded_on: 2026-08-17
---
> **SUPERSEDED 2026-08-17.** ⚠ The steward's rule this file carried is INSTRUMENT-INDEPENDENT and was harvested before this stamp: feedback-bulk-indexing-runs-incrementally-with-readback.md. It applies now to studium-engine corpus work. Superseded for MemPalace operations only.
>
> Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the
> status below as live.


## Context

The 2026-05-01 → 2026-05-03 chamber-library mine on bge-m3+MPS produced 933,576 drawers in storage but unreadable by chromadb 1.5.8 (format error: "Palace is NOT readable by chromadb 1.5.8"). Three days of CPU/MPS lost. Palace preserved at `~/.mempalace/palace.bge-m3-broken-2026-05-04/` for later forensic recovery.

Compounding architectural issue: the local patch in `mempalace/backends/chroma.py` (`_mempalace_embedding_fn`) forced bge-m3 (1024d) for ALL collections globally. This created a hard conflict with the pre-bge-m3 palace (mxbai 384d). Reverted 2026-05-04 to restore baseline functionality on the 131,418 pre-bge-m3 drawers (Obsidian vault, Claude transcripts, etc.). Patch saved at `~/_Dev/mempalace/.local-data/bge-m3-patch-2026-04-30.diff` for reference.

## Steward's rule (verbatim)

> *"go with path A. then we make a choice on how to remine the library incrementally beginning with the most critical texts. I'm not losing days to this again."*

— David Glidden, 2026-05-04

## The rule, expanded

1. **No single multi-day mine of chamber-library ever again.** A failure that's only visible after days of compute is unacceptable. Mines are incremental and validated end-to-end (storage AND search) before committing more compute.

2. **Prioritize by criticality.** The library has many texts; only some are load-bearing for active ARC work. Start with what's actually being used (recently cited / actively being read / referenced in current spec work).

3. **Validate after each tranche.** Storage write + search readback must both succeed before the next tranche starts. The bge-m3 mine's failure mode — storage succeeds, search corrupts silently — must be impossible going forward.

4. **The architectural fix for the global-patch conflict** is per-collection embedding configuration: bge-m3 for `chamber_library` (multilingual content), default mxbai for everything else. The current saved patch forces one global model and is not a viable long-term answer.

## Open questions for the next library work session

1. **Triage**: which library texts are actually load-bearing right now? (probably: Bringhurst, Manutius the Younger, Berger, Athenaeus, Vico, Harrison, the Observer Problem sources, anything cited in active essays — but the steward should confirm)
2. **Forensic on the broken palace**: is the data recoverable, or do we re-mine source files? Cheapest first move: try `mempalace migrate` on `palace.bge-m3-broken-2026-05-04/` (the format error is exactly what `migrate` exists for; it might just work). Status 2026-05-04: dry-run on big palace started but was killed when scope shifted to baseline restore.
3. **Embedding strategy**: continue with bge-m3 for multilingual quality, OR accept default mxbai for English-only criticality? bge-m3 quality non-negotiable per 2026-05-01 decision (chamber-library has critical French + some German texts), so the architectural fix (per-collection embedding) is the right path.
4. **Per-collection embedding patch**: refactor the saved patch in `.local-data/bge-m3-patch-2026-04-30.diff` to apply only when collection name matches `chamber_library*` (or via explicit allow-list). Should be ~10-line change. Test against a 100-file mine before any larger commitment.

## Anti-patterns to refuse

- "Just re-run the mine, it'll probably work this time" — three days lost to that posture.
- "Skip the dry-run / verification step, we already know it works" — the bge-m3 mine wrote 933k drawers that look ready by `mempalace_status` and fail on first search. Storage success ≠ recall success.
- "It's only multilingual, English content might still work without bge-m3" — quality assessment was already made: chamber-library has critical French + German content. Don't relitigate.

## Status 2026-05-04

Baseline restored. Library re-mine strategy parked until steward bandwidth + decision on triage criteria. Forensic on broken big palace is a candidate first move (cheap if it works).