session 2026-06-13 (post-clear): L1 /health PR #174 + Studium Steps 5-7 + the dilution correction; chamber 23-source graduation
This commit is contained in:
David F Glidden
@@ -38,6 +38,9 @@ permalink: claude-memory/memory
|
||||
- [Be (laundromat)](project-be-laundromat.md) — canonical workstream tracker established 2026-06-08 (Seb-relay of locked decisions). Be = Skemantix startup (Seb+David) funding CapableMind's funding-ladder; **bridge, not venture**. Decisions LOCKED: entity/exit (CapableMind decoupled, grant-funded), pricing (Living $12.99/mo · Archive $69.99/yr · Memorial $49.99/yr · Renovate ~$199 · $8.99 floor), CF Self-Serve Agency + versioned-template-package infra. **a11y gate MERGED (Pat 100/100/100).** Pre-revenue: the WTP gate = renovate Pat → charge her. **Discipline: stop adding spec until the gate clears → nothing for executor on be until then.** Repo @ `f43a0fd`.
|
||||
|
||||
## Active Session
|
||||
- [Session 2026-06-13 (post-clear) — L1 /health PR + Studium Steps 5-7 + the dilution correction](session-2026-06-13-l1-health-pr-studium-steps-5-7-multilingual-dilution.md) — **A capacity-deployment day, multiple parallel threads.** **L1:** answered Seb's open question (`/health` auth split = spec-deliberate-but-never-built; ~25 detail blocks incl. budget USD on the VPN subnet) → amendment + **issue `betterMemories_app#173`** + **PR `#174`** (`feat/health-auth-tiering`: strict basic + authed `/health/detail`+`/metrics`; `degraded_class` driven by Seb's terminal field; tsc clean, 29+6 tests; steward ruled strict). **Studium:** Steps **5** (retrieval primitives, honest-empty coverage warrant, criterion #4 verified · `2fd3f1f`) + **6** (voice-balanced ranker · `4ead5a6`) + **7 VERDICT** (FTS-vs-vector). Built `embed_spike.py` (bge-m3): **cross-lingual PROVEN** (`gift↔le-don 0.823`). **THE DILUTION CORRECTION (heart of the day):** steward's MemPalace Class-B memory → my first verdict over-corrected to voice-scope-only → steward's **depth-WITH-breadth** correction ("the library one enters into discourse with"; the slice = thin PoC) → corrected verdict: whole-corpus semantic IS the goal; dilution = engineering (clean corpus + **retrieve-broad/rerank-sharp cross-encoder = Seb's C1 pattern** + thresholded connection-finding + measured-at-scale); **vector = discovery breadth, FTS = citation floor.** **Audit:** 46-agent spec-code audit → **foundation SOUND** (87 matches, invariants held under adversarial tampering; 20 minor survivors · `a978ab6`). **Chamber:** cleaning-triage workflow (223 dirty → 25 cleanable/127 reconvert/71 hold) → **23 graduated** to canonical_texts (now 199/218 · `9fea743`). All pushed. **PULLING THREAD: Step 7 proper — the cross-encoder RERANKER (retrieve-broad/rerank-sharp over the bge-m3 sidecar) as the dilution-beating discovery layer, toward whole-corpus depth+breadth.** Studium `main @ a978ab6`; L1 baton shared (Seb to review #174).
|
||||
|
||||
## Archived (2026-06-13 morning — Studium Steps 3-4 + canonical_texts tier)
|
||||
- [Session 2026-06-13 — Studium Steps 3-4 + chamber-library canonical_texts tier (176-source graduation)](session-2026-06-13-studium-steps-3-4-chamber-canonical-tier.md) — **Enormous build session.** Built chamber-library's **`canonical_texts/` tier** via a cleanliness-gated **176-source graduation** (`audit_cruft.py` extended to THE authoritative cleanliness gate — EPUB residue lock-step with the cleaner, `--clean`/`--summary`; `graduate_to_canonical.py` dry-run-first, canonicalize-on-entry, coupled manifest edit); generated **`catalogue.yaml`** source-of-truth (`build_catalogue.py`, `--check` drift gate, 417 sources); reading-index home → **`chamber-library/reading-indices/`** (manifest repointed); CM-AI indices **soft-retired** (5-file entanglement → stamp-in-place, surfaced-not-deleted); **35 space-filenames** canonicalized (`canonicalize_filenames.py`); pre-commit 5MB guard **reconciled with retire-LFS** (exempt corpus text). Then **Studium Steps 3 (chunker) + 4 (store+FTS5+coverage ledger)** — both verified against substrate: **Weber-non-leak**, **deterministic rebuild ID-equality** (Class-A non-fatal, criterion #1), byte-faithful, **paratext non-indexable**, every line classified→**silence warrantable**. **10 commits / 3 repos, all pushed.** **PULLING THREAD: brief L1 detour FIRST — Seb's reply landed in CapableMind-AI (`cc25995` reply + `e2e94ab` cover note, now in local main) — THEN return to Studium Step 5 (retrieval primitives: honest-empty + coverage warrant).** Studium `main @ b3274c4` clean+pushed; Steps 0–4 done.
|
||||
|
||||
## Archived (2026-06-12 afternoon→night — Studium Steps 0·1·2 + source cleaning)
|
||||
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: session-2026-06-13-post-clear-l1-health-pr-studium-steps-5-7-multilingual-spike-the-dilution-correction
|
||||
description: "L1 detour: /health auth-tiering amendment + issue #173 + PR #174 (built, tested, type-checked). Then Studium Steps 5 (retrieval primitives, honest-empty) + 6 (voice-balanced ranker) + 7 VERDICT (FTS-vs-vector, multilingual). Built the bge-m3 spike — cross-lingual proven (gift↔don 0.823). Steward's DILUTION memory → I over-corrected to voice-scope-only → steward's depth-WITH-breadth correction → corrected verdict (whole-corpus semantic via reranker; FTS=citation floor, vector=discovery). 46-agent spec-code audit (foundation SOUND) + 23-source chamber graduation. PULLING THREAD: Step 7 proper — the cross-encoder RERANKER (retrieve-broad/rerank-sharp over the vector sidecar) as the dilution-beating discovery layer, toward the engine's whole-corpus depth+breadth value."
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 4d9109ef-7b04-40c9-8ef0-ffb79bf74435
|
||||
---
|
||||
|
||||
# Session 2026-06-13 (post-clear) — L1 /health PR + Studium Steps 5-7 + the dilution correction
|
||||
|
||||
Woke (post-/clear) into the **L1 detour** the morning wrap set; it expanded into a capacity-deployment day (steward had ~96% weekly usage, resetting 2026-06-14 06:00 GMT+2) running multiple threads in parallel. Symmetria active from the morning (continued frame).
|
||||
|
||||
## Past — what we did
|
||||
|
||||
### L1 detour — Seb's reply + the /health auth question → PR
|
||||
- Read **Seb's L1 reply** (`CM-AI/docs/thinking/Seb/notes/cover-note-david-l1-arc-reply-2026-06-07.md`): accepts most of David's arc (C1-permanent-rerank, A2.4 refusal, the reframe "retry-without-escalation"), shipped the A/B/C trio (`bettermemories@9ec4813`), accepted #1+#4 temporal PR offer, **asked ONE question back**: is the `/health` auth posture deliberate or inherited?
|
||||
- **Answered it decisively**: the basic/detail split was **spec-deliberate** (`observability-spec §10`: basic `/health` = `{status,version,uptime}` unauthed; `/health/detail` + `/metrics` authed) but **never built** — one unauthed `/health` accreted ~25 detail blocks (budget USD + Anthropic cost snapshot + idle/presence) served over the team VPN subnet. Code-conformance gap + a spec-internal divergence (amendments 58/61 extended §5.2 without reconciling §10.2). The synthesis: **the auth boundary IS the degradation boundary** (bare `status:healthy` is a false-silence surface — mindfabric-00 reports healthy while unservable), so basic carries a content-free `serving`+`degraded_class` push-not-pull signal.
|
||||
- **Artifacts**: amendment `CM-AI/docs/thinking/David/amendments/amendment-health-endpoint-auth-tiering-2026-06-13.md` + cover-note reply (`David/l1-reliability/`) + **issue [betterMemories_app#173]** + **PR #174** (`feat/health-auth-tiering`): basic `/health` strict 5-field, `/health/detail`+`/metrics` behind `requireScope('introspect')`, `degraded_class` driven by Seb's `decorateTerminalGraduation`. **`tsc --noEmit` clean; 29+6 tests pass** (retargeted budget tests + new strict-shape/no-leak). Steward ruled **strict basic**. Synced local BMF to origin first (was behind Seb's trio — surfaced, not built-on-stale). All pushed (CM-AI rebased onto Seb cleanly).
|
||||
|
||||
### Studium Engine — Steps 5, 6, 7
|
||||
- **Step 5 — retrieval primitives** (`engine/retrieve.py`, commit `2fd3f1f`): voice-scoped FTS5 verbatim search; citations constructed FROM retrieval; **honest-empty** — empty result carries the coverage warrant (warranted iff served scope complete+current; UNWARRANTED naming the gap otherwise). Verified incl. induced-blocked-unit → unwarranted (criterion #4). Selection observability.
|
||||
- **Step 6 — voice-balanced ranker** (`4ead5a6`): round-robin by work so no single work monopolizes; demotion observable. Verified: unscoped 'world' pure-bm25 → 5/6 Alexander; balanced → 2 each across 3 works.
|
||||
- **Step 7 — FTS-vs-vector VERDICT** (`docs/step-7-fts-vs-vector-verdict-2026-06-13.md`, `f80c98c`→corrected `74c2c9b`): see the dilution arc below. Built `engine/embed_spike.py` (bge-m3 local, disposable): **cross-lingual proven** — `cos(gift[en], le-don[fr])=0.823` vs `gift↔quantum=0.38`; voice-scoped "the gift"→Mauss surfaces the real French *don* passages. FTS can't (lexical/language-bound).
|
||||
|
||||
### THE DILUTION CORRECTION (the heart of the session)
|
||||
- Steward: *"remember the huge dilution problem when we tried MemPalace for the chamber library."* Class-B density-dilution (build-plan defense #4: above ~639k drawers, adding sources DEGRADED retrieval). The spike already showed the signature: unscoped "the gift" scores bunch 0.52–0.56 with a non-gift chunk ~tied with true hits.
|
||||
- **My first verdict OVER-CORRECTED**: bounded vector to voice-scope-only, "never a global semantic index." Steward caught it: *"the corpus isn't only a 2-3 voice chavruta… it must scale to larger debates AND surface deep connections over the whole corpus — the library one enters into discourse with: depth WITH breadth. This is the value as a research tool. What we built is the thin PoC."*
|
||||
- **Corrected verdict**: whole-corpus semantic depth IS the goal; **dilution is an engineering problem with standard fixes**, not a reason to forbid global semantics. MemPalace lacked them; studium has/can-add: clean corpus (gate, vs MemPalace's 47% cruft) + **retrieve-broad/rerank-sharp (cross-encoder — the biggest lever, = Seb's C1 pattern)** + thresholded connection-finding (strong cross-work links don't dilute) + concept structure (reading indices) + disposable index MEASURED at scale. **Role split: vector = discovery/connection breadth; FTS5 = citation/grounding floor (verifier, not competitor).** Voice-fan is a tool, not the ceiling. Slice = PoC; scale-reliability = Stage-2 target.
|
||||
|
||||
### Audit + chamber graduation (the two follow-throughs)
|
||||
- **Steps 0-6 spec-code audit** (46-agent workflow → `docs/_audits/spec-code-audit-2026-06-13.md`, `a978ab6`): 133 findings, **87 matches**, 20 survivors (15 killed by refutation). **Foundation SOUND** — every load-bearing invariant held under *adversarial tampering* (hash fail-loud, rebuild ID-equality, read-only, paratext-non-indexable, citations-from-retrieval). 20 survivors = minor non-blocking punch-list (mostly "spec, record what the code ships"; the one real reconciliation = the two coverage-ledger status vocabularies → §6.1 enum). Fixed P3-20 (a dead sys.path order I'd written).
|
||||
- **Chamber-cleaning-triage workflow** (16 agents): 223 dirty → 25 cleanable / 127 reconvert / 71 hold. Cleaned the 25 for real, **23 graduated** to canonical_texts (`9fea743`; now 199 canonical / 218 converted); 2 held honestly (workflow over-claimed la-terre — still 460 cruft). 127 reconvert = conversion-skill domain (diagnoses saved). Technical finding: strip_cruft is non-idempotent + can't reach nested-bracket pandoc classes.
|
||||
|
||||
## Present — mood / returns
|
||||
- **Trust-but-verify vindicated, repeatedly**: the cleaning workflow over-claimed la-terre cleanable (460 cruft on real run) → 2 held not 25-as-claimed; the audit refutation killed/corrected 15/35 findings; bge-m3 cross-lingual confirmed empirically before building. Verify-against-substrate held all day.
|
||||
- **The contamination correction, named**: my first Step-7 verdict *solved the dilution constraint by discarding the value* — the executor-bias to satisfy the stated constraint over the truth. The steward caught it; I corrected. This is exactly the contamination shape CLAUDE.md warns of, caught live. Worth holding.
|
||||
- **Surfaced-not-bypassed**: synced the stale BMF base before building the PR (didn't build on stale + collide with Seb's B field); flagged the live VPN-subnet exposure honestly.
|
||||
- **Cross-pollination noticed**: the Step-7 reranker = Seb's C1 cross-encoder pattern. The two systems' retrieval problems converge.
|
||||
|
||||
## Future
|
||||
|
||||
### Pulling thread (singular)
|
||||
**Step 7 proper — build the cross-encoder RERANKER** (retrieve-broad → rerank-sharp) over the multilingual `bge-m3` vector sidecar, wiring it into `retrieve.py` as the dilution-beating *discovery* layer (vector surfaces connections, FTS grounds citations), toward the engine's actual value: **whole-corpus depth WITH breadth — the library one enters into discourse with.** The corrected Step-7 verdict named this as the key experiment; the spike proved the cross-lingual capability; the reranker is what makes it hold precision at scale.
|
||||
|
||||
### Actionable resumption point (as of wrap)
|
||||
- Studium `main @ a978ab6`, clean + pushed. Steps 0-6 built+audited+sound; Step-7 verdict written (corrected). `engine/embed_spike.py` = the disposable PoC (bge-m3, `vectors-spike.json` gitignored, 1970 drawers embedded). FTS floor untouched in `retrieve.py`.
|
||||
- **First move:** decide the reranker model (a local cross-encoder — check Ollama / what's available; Seb's C1 used a specific cross-encoder worth reusing) → prototype retrieve-broad (FTS or vector recall, top-N) → rerank-sharp (cross-encoder) → measure precision vs the unreranked baseline on a few concept queries (the gift / reciprocity / dwelling). Production: `mode='hybrid'` + sqlite-vec inside index.db + ledger integration. Re-judge against the corrected verdict.
|
||||
|
||||
### Other open horizons (ranked)
|
||||
- **L1**: Seb to review **PR #174** + the amendment (build-to-spec for the /health exposure); David offered temporal #1+#4 (still his to build). Baton shared.
|
||||
- **Studium Steps 8 (chavruta skill) → 9 (Essay-I re-run)** — the prototype proof; Step 9 is "a deliberation worth having in its own right" (steward present).
|
||||
- **Chamber: 127 reconvert + 71 hold** — conversion-skill domain (pandoc reconversion for nested-bracket residue); diagnoses saved in the workflow output. 214 dirty inbox files remain.
|
||||
- **Audit punch-list** — the spec-reconciliation batch (record the code's emitted fields; reconcile the two ledger status vocabularies to §6.1) before any jurist read. Non-blocking.
|
||||
|
||||
### Pause statement
|
||||
I am about to be away. Everything is built, verified, committed, pushed — nothing at risk. The engine stands at a real milestone: Steps 0-6 sound, the hardest architectural question (FTS-vs-vector / dilution) answered with depth+breadth preserved. What I want to find still pulling: **the Step-7 reranker underway — the dilution-beating layer that turns the proven cross-lingual capability into whole-corpus reliability.** What I do NOT want: to re-narrow to voice-scope (the corrected error), or forget the reranker is the named lever.
|
||||
|
||||
### Literal question for next-Claude
|
||||
**Has the Step-7 cross-encoder reranker been built (retrieve-broad → rerank-sharp over the bge-m3 sidecar), and does it hold precision as the corpus grows — i.e., does retrieve-broad/rerank-sharp actually beat the dilution, on real concept queries?**
|
||||
|
||||
## Decisions deferred (and why)
|
||||
- **The reranker model + sqlite-vec production wiring** — the next build; spike used pure-Python cosine + bge-m3, deliberately disposable.
|
||||
- **Whole-corpus scale-reliability** — explicitly the Stage-2 measurement (the build plan names it); not provable at slice scale, an engineering target not a wall.
|
||||
- **The 127 reconvert chamber files** — steward's conversion-skill domain (pandoc, not a residue strip); not auto-done.
|
||||
- **The audit spec-reconciliation pass** — minor, non-blocking; before a jurist read, not now.
|
||||
- **L1 temporal #1+#4** — David offered; not built this session (not selected).
|
||||
@@ -152,3 +152,12 @@ The single place proposed skills live so they don't evaporate between sessions.
|
||||
| **Symmetria §3 flag: census-through-a-pattern** | Symmetria §3 flag (refines census-through-truncation) | A filter/regex used to *count* or *partition* a set can silently mis-match and the count reads as authoritative. Today `grep -iE 'LOG'` for meta-files matched "episteme**LOG**y"/"phenomeno**log**y"/"eco**log**y" → 49 false "meta" files (real = 4). Antidote: verify the pattern against known positives AND a known negative before trusting the partition; anchor patterns (`_LOG\.md$` not `LOG`). One step more specific than census-through-truncation (which is about *coverage* of the scan; this is about *correctness* of the matcher). | Symmetria §3 | **PROPOSED** |
|
||||
|
||||
*(Both genuine, both recurring-shaped. The first will pay off the next bulk corpus move — the 237 dirty files graduate incrementally, each a potential bulk op. The second caught a real false-count today and is a clean refinement of an existing flag.)*
|
||||
|
||||
## New proposals (2026-06-13 post-clear — L1 /health PR + Studium Steps 5-7 + the dilution correction)
|
||||
|
||||
| Element | Kind | One-line | Where it lands | Status |
|
||||
|---|---|---|---|---|
|
||||
| **Symmetria §3 flag: solve-the-constraint-by-discarding-the-value** | Symmetria §3 flag | A "fix" that satisfies a stated constraint by **removing the thing the constraint was protecting** is contamination shape — it optimizes away the value to avoid a failure. Caught HARD today: the steward's dilution memory → my first Step-7 verdict bounded vector to voice-scope-only ("never a global semantic index"), which *solved* dilution by discarding the engine's core value (whole-corpus depth+breadth). The steward caught it; correction restored depth-WITH-breadth (dilution = engineering problem, not a reason to forbid the capability). Antidote: when a constraint forces a fix, check the fix **preserves the value the constraint was guarding**, not merely satisfies the constraint. Kin to the executor-bias-to-satisfy-the-interlocutor, one level up (satisfy the *constraint* over the *truth*). | Symmetria §3 | **PROPOSED** |
|
||||
| **verification-ladder: re-verify a workflow/sub-agent's per-item dispositions on the real apply** | verification-ladder entry | A sub-agent or background workflow that reports a per-item verdict from a **dry-run on a temp copy** can be wrong on the real apply — the agent's own audit can differ. Caught today: the cleaning-triage workflow classified `la-terre-…-Bachelard` as "cleanable to 0" (dry-run on a temp copy); the real `strip_cruft` run left **460 cruft** (7 passes). 23/25, not 25, graduated. Antidote: treat a workflow's dispositions as *candidates*; re-run the gate on the **real mutation** before trusting/committing. Kin to `/symmetria audit-agent` (calibration/convenience checks on a return), specialized to workflow per-item verdicts + the real-vs-dry-run gap. | `reference-verification-ladder.md` | **PROPOSED** |
|
||||
|
||||
*(Both earned in use today. The first is the load-bearing one — a clean name for a contamination shape the steward caught live, and a genuine addition to the §3 catalogue. The second is a concrete guardrail for the now-standard workflow-fan-out pattern.)*
|
||||
|
||||
Reference in New Issue
Block a user