David F GliddenandClaude Opus 5
ef6fa94dcd
The record is updated where information ARRIVES and never where a reader LOOKS.
Three instances found the same day, each with the correct information already
written down elsewhere:
REVIEWED-81 (AUTHORIZED 07-28) -> PENDING-81 still reads open
the ARC open-work register, which -> those 6 trackers still read live
says in its own description that
it supersedes them, with a
per-file verified verdict
register entry A2: "the content -> register entry B3, one page below:
half (B3) is now unblocked + "[OPEN - gated on A2]"
started" (06-17)
The third is the sharpest: both halves sit in the SAME document, so no tooling
gap explains it. ~/CLAUDE.md already rules this under Memory Discipline
("mark the superseded record as superseded"); the rule is being applied to facts
and not to trackers, PENDING items or register entries, which are also facts
about state.
Executed under steward authorization, arrears only:
- 16 trackers stamped `superseded_by:` + `superseded_on:` AND a visible in-body
banner. The banner is the point — metadata alone repeats the defect, since the
finding is about what a reader sees. 9 obsoleted by the 2026-07-07 MemPalace
wind-down; 7 already carrying the register's own verified `DONE-retire`.
- B3's label corrected, with the two-month error left visible rather than
silently amended.
- Nothing moved, renamed or deleted: 381 pointers resolve, 0 dead, 0 mis-authored.
HARVESTED BEFORE STAMPING, and this is why the harvest came first:
`project-mempalace-library-incremental-remine-strategy` carried a steward-verbatim
rule that is instrument-independent — never run a single multi-day index; order by
criticality; gate each tranche on READBACK, not on write-success (933,576 drawers
once stored successfully and could not be read back at all). It lived only inside
a MemPalace tracker, so retiring the instrument would have retired the rule with
it. ~/CLAUDE.md names that exact hazard: "state the obligation first and the
instrument second, or the next retired tool takes a rule down with it." Now
`feedback-bulk-indexing-runs-incrementally-with-readback.md`, and it applies to
studium-engine corpus work.
Held back deliberately: 2 correctly-parked files with named dependencies, 2 named
nowhere (returned to the steward), and 39 trackers unread — recorded as a gap,
not as a finding.
Measurement note: every earlier estimate of tracker staleness was wrong, because
both mtime and git-last-commit were reset across 283 files by the 2026-06-06
normalization sweep. The honest figure excludes it: median 90 days.
Filed as PENDING-142 ADDENDUM 2 per steward direction (fold, do not file anew).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Y6t6qx7cpaCu5xGdD36u4
67 lines
4.5 KiB
Markdown
67 lines
4.5 KiB
Markdown
---
|
|
permalink: claude-memory/project-mempalace-bge-m3-mine-search-broken-2026-05-03
|
|
superseded_by: project-mempalace-winddown.md
|
|
superseded_on: 2026-08-17
|
|
---
|
|
> **SUPERSEDED 2026-08-17.** Diagnostic state of a palace that no longer exists.
|
|
>
|
|
> Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the
|
|
> status below as live.
|
|
|
|
|
|
---
|
|
name: MemPalace bge-m3 mine — drawers stored, search broken (Internal error: Error finding id)
|
|
description: 2026-05-03 diagnostic state. Three days of CPU/MPS time on the bge-m3 chamber-library mine produced 933,576 drawers in storage but functional semantic recall is broken. Real loss for the steward.
|
|
type: project
|
|
originSessionId: 8d9ac240-47df-4578-aec1-a9dd32965027
|
|
---
|
|
## Diagnostic state (2026-05-03 evening)
|
|
|
|
**The good (storage)**
|
|
- `~/.mempalace/palace/chroma.sqlite3` is 6.3 GB
|
|
- `mempalace_status` returns instantly: `total_drawers: 933,576`
|
|
- `mempalace_list_wings` returns instantly: `chamber_library: 933,576`
|
|
- All embeddings are dimension 1024 (bge-m3 confirmed)
|
|
- Two Chroma collections exist: `mempalace_drawers` (1024d), `mempalace_closets` (1024d)
|
|
- Last write to chroma.sqlite3: today (2026-05-03 17:34)
|
|
- Mining process is NOT running — terminated cleanly per machine cold
|
|
|
|
**The bad (search)**
|
|
- `mempalace_search` with any wing-scoped query returns immediately with:
|
|
```
|
|
Search error: Error executing plan: Internal error: Error finding id
|
|
```
|
|
- Tried two queries — both errored:
|
|
- "Athenaeus herons curved necks seed-gathering grouse warblers sukallides fig season" (steward's verbatim test passage)
|
|
- "Bringhurst" (single keyword)
|
|
- Both returned the SAME error string → not a query-content issue
|
|
|
|
**Working hypothesis**
|
|
The bge-m3 + MPS mine produced the storage layer (Chroma's sqlite + embedding tables) but the LanceDB / vector index that backs `search` is incomplete or corrupted. Possible causes:
|
|
- Mine completed embeddings but never finished building/persisting the search index
|
|
- Index segment file got truncated or had write failure mid-build
|
|
- bge-m3 dimension change (was 384 with mxbai, now 1024) requires a fresh index that wasn't fully built
|
|
|
|
## What this cost
|
|
|
|
**Three days of CPU/MPS compute**, per steward's accounting. The steward wrapped the prior session having already deferred the mine for fresh-session restart on MPS to get 5-10x speedup. That work appears to have run to completion on the storage side (drawer count went from 501k @ pause-point on 2026-05-01 to 934k now) — but the recall function the mine was meant to enable is non-functional.
|
|
|
|
This is **not** a "non-fatal" failure. The whole point of the mine was searchable verbatim recall via MCP. Storage without search is useless storage. The contamination-pattern frame: a system that reports "ready" via its own counters (status returns 933k drawers, list_wings returns the wing) but cannot deliver the actual function it was built for.
|
|
|
|
## Important honesty correction
|
|
|
|
When the steward said "40 minutes for mempalace queries — must be slow," that was incorrect framing on my part. Mempalace queries themselves return in milliseconds — `status` and `list_wings` are instant; `search` returns the error string immediately. The 40 minutes was me getting deflected by a cascade of UI regressions (audit-induced) and never coming back to characterize the search failure. **Mempalace is not slow; it is broken.** That distinction matters because the diagnosis differs (slow → optimization; broken → repair).
|
|
|
|
## Possible paths forward (not yet tried)
|
|
|
|
1. **Diagnose first, don't re-mine.** Inspect the Chroma `segments` table + on-disk index files (`palace/<segment-uuid>/`) to see whether the index files are present, what state they're in, and whether the search error refers to a specific missing segment ID.
|
|
2. **Try `--force-rebuild-index`** or equivalent mempalace mine flag (if one exists) — might rebuild the search index without re-embedding. Saves the 3 days.
|
|
3. **Mine to a fresh palace path**, keep the broken one for forensics. Worst case scenario; lose the 3 days.
|
|
4. **File issue upstream** — Seb / mempalace repo. The "Internal error: Error finding id" is opaque enough that it warrants the maintainer's eyes.
|
|
|
|
## Steward signal
|
|
|
|
> "Please take note of the mempalace debugging. Three days of time that mining cost me."
|
|
> "40 minutes. I smell something rotten with mempalace. When you call it it takes forever."
|
|
|
|
The frustration is appropriate. Don't minimize. Don't reach for "still useful" framings. The mine cost real time and the result is not what was promised. |