68 lines
11 KiB
Markdown
68 lines
11 KiB
Markdown
---
|
||
name: session-2026-06-13-post-clear-l1-health-pr-studium-steps-5-7-multilingual-spike-the-dilution-correction
|
||
description: "L1 detour: /health auth-tiering amendment + issue #173 + PR #174 (built, tested, type-checked). Then Studium Steps 5 (retrieval primitives, honest-empty) + 6 (voice-balanced ranker) + 7 VERDICT (FTS-vs-vector, multilingual). Built the bge-m3 spike — cross-lingual proven (gift↔don 0.823). Steward's DILUTION memory → I over-corrected to voice-scope-only → steward's depth-WITH-breadth correction → corrected verdict (whole-corpus semantic via reranker; FTS=citation floor, vector=discovery). 46-agent spec-code audit (foundation SOUND) + 23-source chamber graduation. PULLING THREAD: Step 7 proper — the cross-encoder RERANKER (retrieve-broad/rerank-sharp over the vector sidecar) as the dilution-beating discovery layer, toward the engine's whole-corpus depth+breadth value."
|
||
metadata:
|
||
node_type: memory
|
||
type: project
|
||
originSessionId: 4d9109ef-7b04-40c9-8ef0-ffb79bf74435
|
||
---
|
||
|
||
# Session 2026-06-13 (post-clear) — L1 /health PR + Studium Steps 5-7 + the dilution correction
|
||
|
||
Woke (post-/clear) into the **L1 detour** the morning wrap set; it expanded into a capacity-deployment day (steward had ~96% weekly usage, resetting 2026-06-14 06:00 GMT+2) running multiple threads in parallel. Symmetria active from the morning (continued frame).
|
||
|
||
## Past — what we did
|
||
|
||
### L1 detour — Seb's reply + the /health auth question → PR
|
||
- Read **Seb's L1 reply** (`CM-AI/docs/thinking/Seb/notes/cover-note-david-l1-arc-reply-2026-06-07.md`): accepts most of David's arc (C1-permanent-rerank, A2.4 refusal, the reframe "retry-without-escalation"), shipped the A/B/C trio (`bettermemories@9ec4813`), accepted #1+#4 temporal PR offer, **asked ONE question back**: is the `/health` auth posture deliberate or inherited?
|
||
- **Answered it decisively**: the basic/detail split was **spec-deliberate** (`observability-spec §10`: basic `/health` = `{status,version,uptime}` unauthed; `/health/detail` + `/metrics` authed) but **never built** — one unauthed `/health` accreted ~25 detail blocks (budget USD + Anthropic cost snapshot + idle/presence) served over the team VPN subnet. Code-conformance gap + a spec-internal divergence (amendments 58/61 extended §5.2 without reconciling §10.2). The synthesis: **the auth boundary IS the degradation boundary** (bare `status:healthy` is a false-silence surface — mindfabric-00 reports healthy while unservable), so basic carries a content-free `serving`+`degraded_class` push-not-pull signal.
|
||
- **Artifacts**: amendment `CM-AI/docs/thinking/David/amendments/amendment-health-endpoint-auth-tiering-2026-06-13.md` + cover-note reply (`David/l1-reliability/`) + **issue [betterMemories_app#173]** + **PR #174** (`feat/health-auth-tiering`): basic `/health` strict 5-field, `/health/detail`+`/metrics` behind `requireScope('introspect')`, `degraded_class` driven by Seb's `decorateTerminalGraduation`. **`tsc --noEmit` clean; 29+6 tests pass** (retargeted budget tests + new strict-shape/no-leak). Steward ruled **strict basic**. Synced local BMF to origin first (was behind Seb's trio — surfaced, not built-on-stale). All pushed (CM-AI rebased onto Seb cleanly).
|
||
|
||
### Studium Engine — Steps 5, 6, 7
|
||
- **Step 5 — retrieval primitives** (`engine/retrieve.py`, commit `2fd3f1f`): voice-scoped FTS5 verbatim search; citations constructed FROM retrieval; **honest-empty** — empty result carries the coverage warrant (warranted iff served scope complete+current; UNWARRANTED naming the gap otherwise). Verified incl. induced-blocked-unit → unwarranted (criterion #4). Selection observability.
|
||
- **Step 6 — voice-balanced ranker** (`4ead5a6`): round-robin by work so no single work monopolizes; demotion observable. Verified: unscoped 'world' pure-bm25 → 5/6 Alexander; balanced → 2 each across 3 works.
|
||
- **Step 7 — FTS-vs-vector VERDICT** (`docs/step-7-fts-vs-vector-verdict-2026-06-13.md`, `f80c98c`→corrected `74c2c9b`): see the dilution arc below. Built `engine/embed_spike.py` (bge-m3 local, disposable): **cross-lingual proven** — `cos(gift[en], le-don[fr])=0.823` vs `gift↔quantum=0.38`; voice-scoped "the gift"→Mauss surfaces the real French *don* passages. FTS can't (lexical/language-bound).
|
||
|
||
### THE DILUTION CORRECTION (the heart of the session)
|
||
- Steward: *"remember the huge dilution problem when we tried MemPalace for the chamber library."* Class-B density-dilution (build-plan defense #4: above ~639k drawers, adding sources DEGRADED retrieval). The spike already showed the signature: unscoped "the gift" scores bunch 0.52–0.56 with a non-gift chunk ~tied with true hits.
|
||
- **My first verdict OVER-CORRECTED**: bounded vector to voice-scope-only, "never a global semantic index." Steward caught it: *"the corpus isn't only a 2-3 voice chavruta… it must scale to larger debates AND surface deep connections over the whole corpus — the library one enters into discourse with: depth WITH breadth. This is the value as a research tool. What we built is the thin PoC."*
|
||
- **Corrected verdict**: whole-corpus semantic depth IS the goal; **dilution is an engineering problem with standard fixes**, not a reason to forbid global semantics. MemPalace lacked them; studium has/can-add: clean corpus (gate, vs MemPalace's 47% cruft) + **retrieve-broad/rerank-sharp (cross-encoder — the biggest lever, = Seb's C1 pattern)** + thresholded connection-finding (strong cross-work links don't dilute) + concept structure (reading indices) + disposable index MEASURED at scale. **Role split: vector = discovery/connection breadth; FTS5 = citation/grounding floor (verifier, not competitor).** Voice-fan is a tool, not the ceiling. Slice = PoC; scale-reliability = Stage-2 target.
|
||
|
||
### Audit + chamber graduation (the two follow-throughs)
|
||
- **Steps 0-6 spec-code audit** (46-agent workflow → `docs/_audits/spec-code-audit-2026-06-13.md`, `a978ab6`): 133 findings, **87 matches**, 20 survivors (15 killed by refutation). **Foundation SOUND** — every load-bearing invariant held under *adversarial tampering* (hash fail-loud, rebuild ID-equality, read-only, paratext-non-indexable, citations-from-retrieval). 20 survivors = minor non-blocking punch-list (mostly "spec, record what the code ships"; the one real reconciliation = the two coverage-ledger status vocabularies → §6.1 enum). Fixed P3-20 (a dead sys.path order I'd written).
|
||
- **Chamber-cleaning-triage workflow** (16 agents): 223 dirty → 25 cleanable / 127 reconvert / 71 hold. Cleaned the 25 for real, **23 graduated** to canonical_texts (`9fea743`; now 199 canonical / 218 converted); 2 held honestly (workflow over-claimed la-terre — still 460 cruft). 127 reconvert = conversion-skill domain (diagnoses saved). Technical finding: strip_cruft is non-idempotent + can't reach nested-bracket pandoc classes.
|
||
|
||
## Present — mood / returns
|
||
- **Trust-but-verify vindicated, repeatedly**: the cleaning workflow over-claimed la-terre cleanable (460 cruft on real run) → 2 held not 25-as-claimed; the audit refutation killed/corrected 15/35 findings; bge-m3 cross-lingual confirmed empirically before building. Verify-against-substrate held all day.
|
||
- **The contamination correction, named**: my first Step-7 verdict *solved the dilution constraint by discarding the value* — the executor-bias to satisfy the stated constraint over the truth. The steward caught it; I corrected. This is exactly the contamination shape CLAUDE.md warns of, caught live. Worth holding.
|
||
- **Surfaced-not-bypassed**: synced the stale BMF base before building the PR (didn't build on stale + collide with Seb's B field); flagged the live VPN-subnet exposure honestly.
|
||
- **Cross-pollination noticed**: the Step-7 reranker = Seb's C1 cross-encoder pattern. The two systems' retrieval problems converge.
|
||
|
||
## Future
|
||
|
||
### Pulling thread (singular)
|
||
**Step 7 proper — build the cross-encoder RERANKER** (retrieve-broad → rerank-sharp) over the multilingual `bge-m3` vector sidecar, wiring it into `retrieve.py` as the dilution-beating *discovery* layer (vector surfaces connections, FTS grounds citations), toward the engine's actual value: **whole-corpus depth WITH breadth — the library one enters into discourse with.** The corrected Step-7 verdict named this as the key experiment; the spike proved the cross-lingual capability; the reranker is what makes it hold precision at scale.
|
||
|
||
### Actionable resumption point (as of wrap)
|
||
- Studium `main @ a978ab6`, clean + pushed. Steps 0-6 built+audited+sound; Step-7 verdict written (corrected). `engine/embed_spike.py` = the disposable PoC (bge-m3, `vectors-spike.json` gitignored, 1970 drawers embedded). FTS floor untouched in `retrieve.py`.
|
||
- **First move:** decide the reranker model (a local cross-encoder — check Ollama / what's available; Seb's C1 used a specific cross-encoder worth reusing) → prototype retrieve-broad (FTS or vector recall, top-N) → rerank-sharp (cross-encoder) → measure precision vs the unreranked baseline on a few concept queries (the gift / reciprocity / dwelling). Production: `mode='hybrid'` + sqlite-vec inside index.db + ledger integration. Re-judge against the corrected verdict.
|
||
|
||
### Other open horizons (ranked)
|
||
- **L1**: Seb to review **PR #174** + the amendment (build-to-spec for the /health exposure); David offered temporal #1+#4 (still his to build). Baton shared.
|
||
- **Studium Steps 8 (chavruta skill) → 9 (Essay-I re-run)** — the prototype proof; Step 9 is "a deliberation worth having in its own right" (steward present).
|
||
- **Chamber: 127 reconvert + 71 hold** — conversion-skill domain (pandoc reconversion for nested-bracket residue); diagnoses saved in the workflow output. 214 dirty inbox files remain.
|
||
- **Audit punch-list** — the spec-reconciliation batch (record the code's emitted fields; reconcile the two ledger status vocabularies to §6.1) before any jurist read. Non-blocking.
|
||
|
||
### Pause statement
|
||
I am about to be away. Everything is built, verified, committed, pushed — nothing at risk. The engine stands at a real milestone: Steps 0-6 sound, the hardest architectural question (FTS-vs-vector / dilution) answered with depth+breadth preserved. What I want to find still pulling: **the Step-7 reranker underway — the dilution-beating layer that turns the proven cross-lingual capability into whole-corpus reliability.** What I do NOT want: to re-narrow to voice-scope (the corrected error), or forget the reranker is the named lever.
|
||
|
||
### Literal question for next-Claude
|
||
**Has the Step-7 cross-encoder reranker been built (retrieve-broad → rerank-sharp over the bge-m3 sidecar), and does it hold precision as the corpus grows — i.e., does retrieve-broad/rerank-sharp actually beat the dilution, on real concept queries?**
|
||
|
||
## Decisions deferred (and why)
|
||
- **The reranker model + sqlite-vec production wiring** — the next build; spike used pure-Python cosine + bge-m3, deliberately disposable.
|
||
- **Whole-corpus scale-reliability** — explicitly the Stage-2 measurement (the build plan names it); not provable at slice scale, an engineering target not a wall.
|
||
- **The 127 reconvert chamber files** — steward's conversion-skill domain (pandoc, not a residue strip); not auto-done.
|
||
- **The audit spec-reconciliation pass** — minor, non-blocking; before a jurist read, not now.
|
||
- **L1 temporal #1+#4** — David offered; not built this session (not selected).
|