David F GliddenandClaude Opus 5
ef6fa94dcd
The record is updated where information ARRIVES and never where a reader LOOKS.
Three instances found the same day, each with the correct information already
written down elsewhere:
REVIEWED-81 (AUTHORIZED 07-28) -> PENDING-81 still reads open
the ARC open-work register, which -> those 6 trackers still read live
says in its own description that
it supersedes them, with a
per-file verified verdict
register entry A2: "the content -> register entry B3, one page below:
half (B3) is now unblocked + "[OPEN - gated on A2]"
started" (06-17)
The third is the sharpest: both halves sit in the SAME document, so no tooling
gap explains it. ~/CLAUDE.md already rules this under Memory Discipline
("mark the superseded record as superseded"); the rule is being applied to facts
and not to trackers, PENDING items or register entries, which are also facts
about state.
Executed under steward authorization, arrears only:
- 16 trackers stamped `superseded_by:` + `superseded_on:` AND a visible in-body
banner. The banner is the point — metadata alone repeats the defect, since the
finding is about what a reader sees. 9 obsoleted by the 2026-07-07 MemPalace
wind-down; 7 already carrying the register's own verified `DONE-retire`.
- B3's label corrected, with the two-month error left visible rather than
silently amended.
- Nothing moved, renamed or deleted: 381 pointers resolve, 0 dead, 0 mis-authored.
HARVESTED BEFORE STAMPING, and this is why the harvest came first:
`project-mempalace-library-incremental-remine-strategy` carried a steward-verbatim
rule that is instrument-independent — never run a single multi-day index; order by
criticality; gate each tranche on READBACK, not on write-success (933,576 drawers
once stored successfully and could not be read back at all). It lived only inside
a MemPalace tracker, so retiring the instrument would have retired the rule with
it. ~/CLAUDE.md names that exact hazard: "state the obligation first and the
instrument second, or the next retired tool takes a rule down with it." Now
`feedback-bulk-indexing-runs-incrementally-with-readback.md`, and it applies to
studium-engine corpus work.
Held back deliberately: 2 correctly-parked files with named dependencies, 2 named
nowhere (returned to the steward), and 39 trackers unread — recorded as a gap,
not as a finding.
Measurement note: every earlier estimate of tracker staleness was wrong, because
both mtime and git-last-commit were reset across 283 files by the 2026-06-06
normalization sweep. The honest figure excludes it: median 90 days.
Filed as PENDING-142 ADDENDUM 2 per steward direction (fold, do not file anew).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Y6t6qx7cpaCu5xGdD36u4
48 lines
2.9 KiB
Markdown
48 lines
2.9 KiB
Markdown
---
|
|
name: feedback-bulk-indexing-runs-incrementally-with-readback
|
|
description: "Never run a single multi-day indexing/mining job. Go in tranches, ordered by criticality, and prove each tranche by reading back what was written before starting the next. Steward rule, 2026-05-04, after three days of compute produced unreadable output."
|
|
metadata:
|
|
node_type: memory
|
|
type: feedback
|
|
originSessionId: 81f817e9-4016-4867-ae16-a486f1e8fb46
|
|
modified: 2026-08-17T13:20:31.039Z
|
|
---
|
|
|
|
Never run a bulk index/mine/embed job as one long pass. **Tranches, ordered by
|
|
criticality, with a write *and readback* check after each** before the next begins.
|
|
|
|
Steward, verbatim, 2026-05-04:
|
|
|
|
> *"go with path A. then we make a choice on how to remine the library incrementally
|
|
> beginning with the most critical texts. I'm not losing days to this again."*
|
|
|
|
**Why:** a failure only visible after days of compute is unacceptable, and it is the
|
|
*normal* outcome rather than the unlucky one. The 2026-05-01 → 05-03 chamber-library run
|
|
spent three days of CPU/MPS and wrote **933,576 drawers that stored successfully and
|
|
could not be read back at all** — storage reported success, search returned `Internal
|
|
error: Error finding id`. A run that writes without ever reading back cannot tell
|
|
"working" from "producing garbage at scale", so its own progress reporting is
|
|
uninformative. An earlier attempt burned nine hours to reach 30 of 371 files.
|
|
|
|
**How to apply:**
|
|
- **Tranche it.** No single job whose failure is invisible until it ends. Size the
|
|
tranche so a wasted one costs an hour, not a weekend.
|
|
- **Order by criticality.** Index what is actually load-bearing for live work first, so
|
|
the useful half exists even if the run is abandoned midway.
|
|
- **Readback is the gate, not the write.** Storage success is not the property you want;
|
|
*retrieval of what you stored* is. Query each tranche before starting the next — this
|
|
is the [[reference-verification-ladder]]'s discrimination rule applied to indexing:
|
|
a check that only exercises the write path cannot see a broken read path.
|
|
- **Watch for a global setting applied to a heterogeneous corpus.** The same run was
|
|
compounded by one embedding model being forced across *all* collections; dimensions
|
|
that suit a multilingual scholarly corpus need not suit anything else.
|
|
|
|
⚠ **This rule is instrument-independent and outlived the tool that earned it.** It was
|
|
learned on MemPalace, which was wound down 2026-07-07 ([[project-mempalace-winddown]]) —
|
|
and it was carried *only* inside a MemPalace tracker, so retiring the instrument would
|
|
have retired the rule with it. `~/CLAUDE.md` names that hazard directly: *"state the
|
|
obligation first and the instrument second, or the next retired tool takes a rule down
|
|
with it."* Harvested here 2026-08-17, on the day the superseded tracker was stamped, for
|
|
exactly that reason. **It applies now to the studium engine's corpus mining** — the same
|
|
shape of work, a different tool.
|