---
name: feedback-bulk-indexing-runs-incrementally-with-readback
description: "Never run a single multi-day indexing/mining job. Go in tranches, ordered by criticality, and prove each tranche by reading back what was written before starting the next. Steward rule, 2026-05-04, after three days of compute produced unreadable output."
metadata: 
  node_type: memory
  type: feedback
  originSessionId: 81f817e9-4016-4867-ae16-a486f1e8fb46
  modified: 2026-08-17T13:20:31.039Z
---

Never run a bulk index/mine/embed job as one long pass. **Tranches, ordered by
criticality, with a write *and readback* check after each** before the next begins.

Steward, verbatim, 2026-05-04:

> *"go with path A. then we make a choice on how to remine the library incrementally
> beginning with the most critical texts. I'm not losing days to this again."*

**Why:** a failure only visible after days of compute is unacceptable, and it is the
*normal* outcome rather than the unlucky one. The 2026-05-01 → 05-03 chamber-library run
spent three days of CPU/MPS and wrote **933,576 drawers that stored successfully and
could not be read back at all** — storage reported success, search returned `Internal
error: Error finding id`. A run that writes without ever reading back cannot tell
"working" from "producing garbage at scale", so its own progress reporting is
uninformative. An earlier attempt burned nine hours to reach 30 of 371 files.

**How to apply:**
- **Tranche it.** No single job whose failure is invisible until it ends. Size the
  tranche so a wasted one costs an hour, not a weekend.
- **Order by criticality.** Index what is actually load-bearing for live work first, so
  the useful half exists even if the run is abandoned midway.
- **Readback is the gate, not the write.** Storage success is not the property you want;
  *retrieval of what you stored* is. Query each tranche before starting the next — this
  is the [[reference-verification-ladder]]'s discrimination rule applied to indexing:
  a check that only exercises the write path cannot see a broken read path.
- **Watch for a global setting applied to a heterogeneous corpus.** The same run was
  compounded by one embedding model being forced across *all* collections; dimensions
  that suit a multilingual scholarly corpus need not suit anything else.

⚠ **This rule is instrument-independent and outlived the tool that earned it.** It was
learned on MemPalace, which was wound down 2026-07-07 ([[project-mempalace-winddown]]) —
and it was carried *only* inside a MemPalace tracker, so retiring the instrument would
have retired the rule with it. `~/CLAUDE.md` names that hazard directly: *"state the
obligation first and the instrument second, or the next retired tool takes a rule down
with it."* Harvested here 2026-08-17, on the day the superseded tracker was stamped, for
exactly that reason. **It applies now to the studium engine's corpus mining** — the same
shape of work, a different tool.