David F GliddenandClaude Opus 5
ef6fa94dcd
The record is updated where information ARRIVES and never where a reader LOOKS.
Three instances found the same day, each with the correct information already
written down elsewhere:
REVIEWED-81 (AUTHORIZED 07-28) -> PENDING-81 still reads open
the ARC open-work register, which -> those 6 trackers still read live
says in its own description that
it supersedes them, with a
per-file verified verdict
register entry A2: "the content -> register entry B3, one page below:
half (B3) is now unblocked + "[OPEN - gated on A2]"
started" (06-17)
The third is the sharpest: both halves sit in the SAME document, so no tooling
gap explains it. ~/CLAUDE.md already rules this under Memory Discipline
("mark the superseded record as superseded"); the rule is being applied to facts
and not to trackers, PENDING items or register entries, which are also facts
about state.
Executed under steward authorization, arrears only:
- 16 trackers stamped `superseded_by:` + `superseded_on:` AND a visible in-body
banner. The banner is the point — metadata alone repeats the defect, since the
finding is about what a reader sees. 9 obsoleted by the 2026-07-07 MemPalace
wind-down; 7 already carrying the register's own verified `DONE-retire`.
- B3's label corrected, with the two-month error left visible rather than
silently amended.
- Nothing moved, renamed or deleted: 381 pointers resolve, 0 dead, 0 mis-authored.
HARVESTED BEFORE STAMPING, and this is why the harvest came first:
`project-mempalace-library-incremental-remine-strategy` carried a steward-verbatim
rule that is instrument-independent — never run a single multi-day index; order by
criticality; gate each tranche on READBACK, not on write-success (933,576 drawers
once stored successfully and could not be read back at all). It lived only inside
a MemPalace tracker, so retiring the instrument would have retired the rule with
it. ~/CLAUDE.md names that exact hazard: "state the obligation first and the
instrument second, or the next retired tool takes a rule down with it." Now
`feedback-bulk-indexing-runs-incrementally-with-readback.md`, and it applies to
studium-engine corpus work.
Held back deliberately: 2 correctly-parked files with named dependencies, 2 named
nowhere (returned to the steward), and 39 trackers unread — recorded as a gap,
not as a finding.
Measurement note: every earlier estimate of tracker staleness was wrong, because
both mtime and git-last-commit were reset across 283 files by the 2026-06-06
normalization sweep. The honest figure excludes it: median 90 days.
Filed as PENDING-142 ADDENDUM 2 per steward direction (fold, do not file anew).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Y6t6qx7cpaCu5xGdD36u4
57 lines
4.9 KiB
Markdown
57 lines
4.9 KiB
Markdown
---
|
|
name: MemPalace chamber-library re-mine — incremental strategy, never another multi-day
|
|
loss
|
|
description: After 3 days lost to bge-m3+MPS chamber-library mine that produced unusable
|
|
storage (chromadb format corruption + global embedding patch conflict), steward
|
|
set the rule. Future library mining is incremental, prioritized by criticality,
|
|
never a single multi-day pass.
|
|
type: project
|
|
originSessionId: 45a40b1e-09cd-42bb-a41a-7298f78a8cf0
|
|
permalink: claude-memory/project-mempalace-library-incremental-remine-strategy
|
|
superseded_by: project-mempalace-winddown.md
|
|
superseded_on: 2026-08-17
|
|
---
|
|
> **SUPERSEDED 2026-08-17.** ⚠ The steward's rule this file carried is INSTRUMENT-INDEPENDENT and was harvested before this stamp: feedback-bulk-indexing-runs-incrementally-with-readback.md. It applies now to studium-engine corpus work. Superseded for MemPalace operations only.
|
|
>
|
|
> Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the
|
|
> status below as live.
|
|
|
|
|
|
## Context
|
|
|
|
The 2026-05-01 → 2026-05-03 chamber-library mine on bge-m3+MPS produced 933,576 drawers in storage but unreadable by chromadb 1.5.8 (format error: "Palace is NOT readable by chromadb 1.5.8"). Three days of CPU/MPS lost. Palace preserved at `~/.mempalace/palace.bge-m3-broken-2026-05-04/` for later forensic recovery.
|
|
|
|
Compounding architectural issue: the local patch in `mempalace/backends/chroma.py` (`_mempalace_embedding_fn`) forced bge-m3 (1024d) for ALL collections globally. This created a hard conflict with the pre-bge-m3 palace (mxbai 384d). Reverted 2026-05-04 to restore baseline functionality on the 131,418 pre-bge-m3 drawers (Obsidian vault, Claude transcripts, etc.). Patch saved at `~/_Dev/mempalace/.local-data/bge-m3-patch-2026-04-30.diff` for reference.
|
|
|
|
## Steward's rule (verbatim)
|
|
|
|
> *"go with path A. then we make a choice on how to remine the library incrementally beginning with the most critical texts. I'm not losing days to this again."*
|
|
|
|
— David Glidden, 2026-05-04
|
|
|
|
## The rule, expanded
|
|
|
|
1. **No single multi-day mine of chamber-library ever again.** A failure that's only visible after days of compute is unacceptable. Mines are incremental and validated end-to-end (storage AND search) before committing more compute.
|
|
|
|
2. **Prioritize by criticality.** The library has many texts; only some are load-bearing for active ARC work. Start with what's actually being used (recently cited / actively being read / referenced in current spec work).
|
|
|
|
3. **Validate after each tranche.** Storage write + search readback must both succeed before the next tranche starts. The bge-m3 mine's failure mode — storage succeeds, search corrupts silently — must be impossible going forward.
|
|
|
|
4. **The architectural fix for the global-patch conflict** is per-collection embedding configuration: bge-m3 for `chamber_library` (multilingual content), default mxbai for everything else. The current saved patch forces one global model and is not a viable long-term answer.
|
|
|
|
## Open questions for the next library work session
|
|
|
|
1. **Triage**: which library texts are actually load-bearing right now? (probably: Bringhurst, Manutius the Younger, Berger, Athenaeus, Vico, Harrison, the Observer Problem sources, anything cited in active essays — but the steward should confirm)
|
|
2. **Forensic on the broken palace**: is the data recoverable, or do we re-mine source files? Cheapest first move: try `mempalace migrate` on `palace.bge-m3-broken-2026-05-04/` (the format error is exactly what `migrate` exists for; it might just work). Status 2026-05-04: dry-run on big palace started but was killed when scope shifted to baseline restore.
|
|
3. **Embedding strategy**: continue with bge-m3 for multilingual quality, OR accept default mxbai for English-only criticality? bge-m3 quality non-negotiable per 2026-05-01 decision (chamber-library has critical French + some German texts), so the architectural fix (per-collection embedding) is the right path.
|
|
4. **Per-collection embedding patch**: refactor the saved patch in `.local-data/bge-m3-patch-2026-04-30.diff` to apply only when collection name matches `chamber_library*` (or via explicit allow-list). Should be ~10-line change. Test against a 100-file mine before any larger commitment.
|
|
|
|
## Anti-patterns to refuse
|
|
|
|
- "Just re-run the mine, it'll probably work this time" — three days lost to that posture.
|
|
- "Skip the dry-run / verification step, we already know it works" — the bge-m3 mine wrote 933k drawers that look ready by `mempalace_status` and fail on first search. Storage success ≠ recall success.
|
|
- "It's only multilingual, English content might still work without bge-m3" — quality assessment was already made: chamber-library has critical French + German content. Don't relitigate.
|
|
|
|
## Status 2026-05-04
|
|
|
|
Baseline restored. Library re-mine strategy parked until steward bandwidth + decision on triage criteria. Forensic on broken big palace is a candidate first move (cheap if it works). |