---
name: Session 2026-04-09/10 evening — L1 full pipeline audit, systemic fixes roadmap
description: Comprehensive L1 audit (4 parallel agents), 8 issues filed, Effect-TS adoption proposed as substrate. Research folder established. Full diagnostic-to-engineering-roadmap for Seb.
type: project
originSessionId: 662625c1-73a8-4f3d-a7f1-4ed16a33864c
---
## What we did

### Research folder established
Created `CapableMind-AI/docs/thinking/David/research/` for external project studies. Updated routing table in `thinking/David/CLAUDE.md`. Memory saved for future sessions.

### External landscape research
- **GitHub trending survey** — evaluated 10 trending projects for CapableMind relevance
- **Hermes Agent study** (`2026-04-09-hermes-agent-study.md`) — trajectory-to-training pipeline, skill architecture (conditional activation), memory manager (provider plugin with single-provider constraint). Key insight: trajectory capture as training byproduct.
- **MemPalace study** (`2026-04-08-mempalace-study.md`) — operational assessment of our installed tool. 96.6% recall with vector-only search validates BMF's logchain philosophy. No governance layer, no developmental model — BMF goes beyond it. KG is empty (0 entities).

### L1 recall path audit
- **Recall path audit** (`2026-04-09-l1-recall-path-audit.md`) — full code trace of query entry → results. Corrected 3 stale assumptions from PENDING-5: query router already has intent-based fast paths, classification confidence already threaded through (Amendment 61), stub embeddings already blocked (#15 fix). Real problems: Ollama query SPOF, BM25 broken in SurrealDB, normalization aggressive.
- Filed **#124** (Ollama silent recall failure), **#125** (BM25 broken), **#126** (normalization)

### L1 pipeline landscape
- **Pipeline landscape** (`2026-04-09-l1-pipeline-landscape.md`) — surveyed tools for each problem category:
  - **LanceDB** recommended for vector storage (embedded Rust/Node, Tantivy FTS, no compaction stalls)
  - **@huggingface/transformers** recommended for in-process embedding (eliminates Ollama SPOF)
  - **p-queue** recommended for dispatch backpressure (later superseded by Effect structured concurrency)
  - Graphiti (temporal fact invalidation pattern), Mem0 (conflict resolution), XState (state machines) noted as study-only

### Full pipeline audit (4 parallel agents)
- **Full audit** (`2026-04-09-l1-full-pipeline-audit.md`) — most significant work of the session. Four agents read the entire codebase in parallel:
  1. **March findings verification**: 13/27 fixed, 5 partially fixed, 9 improved with logging. Fire-and-forget 22→19.
  2. **Write path**: Logchain solid, classification never drops events, module isolation works. HIGH: vector battery skip loses events permanently, entity module swallows all pipeline errors.
  3. **Replay/bootstrap/infra**: CRITICAL: modules report `ready` during replay (setReadyPartial/setReady are dead code, coverage annotations never wired). Frozen latency baseline still not fixed. Bug D (idle stall) IS fixed via activity lease.
  4. **Module internals**: Temporal query errors silently return empty. Training graduation still fire-and-forget. ~75 bare catch blocks, ~15 architecturally significant. In-memory state loss universal (anomaly: 14-day gap after restart).
- Filed **#133** (CRITICAL: readiness lie), **#134** (entity error swallowing), **#135** (temporal query silent failure), **#136** (vector battery skip)

### Systemic fixes roadmap
- **Systemic fixes** (`2026-04-10-l1-systemic-fixes.md`) — THE document for Seb. Maps every finding to a systemic solution:
  - **Effect-TS** as substrate: typed errors, structured concurrency, Layers for DI, supervised fibers, built-in tracing. Adopted because team is scaling — do it once correctly. Strangler-fig adoption over 4 weeks.
  - **Position-based readiness**: derived from stream position, not asserted. No new deps.
  - **better-sqlite3 snapshots**: module state survives restart. Already in dep tree.
  - **LanceDB + transformers.js**: decouple vector from SurrealDB, embedding from Ollama.

## What we decided — and why

1. **Research folder in thinking/David/research/** — David is doing increasing landscape scanning. Needs a permanent discoverable home, not ephemeral session notes.

2. **Effect-TS over neverthrow** — Initially recommended neverthrow (simpler, solves only error handling). David pushed back: "We are a two person team TODAY, but we are ramping this up to scale." The prime directive demands doing it once. Effect solves errors + concurrency + DI + tracing + resource lifecycle in one framework. neverthrow would mean adopting a second tool later. Effect is the right long-term bet despite higher upfront cost.

3. **LanceDB over sqlite-vec** — LanceDB has built-in Tantivy FTS (real BM25), Arrow-based columnar storage (no compaction), native Node.js bindings. sqlite-vec is simpler but caps at ~100K vectors and needs separate FTS5. Given scaling intent, LanceDB is proportionate.

4. **Full audit before proposing fixes** — David asked "what's your confidence?" on the recall diagnosis. I admitted it was based on stale PENDING-5 data. David insisted on thorough audit first. The audit corrected multiple stale assumptions and found the CRITICAL readiness lie that wasn't visible from the recall path alone.

## What's unresolved

- **None of the 8 filed issues have responses from Seb yet** — all filed today, Seb is active but busy with other priorities (Amendment 62 / BetterBridge)
- **DN-RET-02 practitioner questions** — 5 questions about score memorization under performance conditions, still unanswered by steward
- **Effect-TS adoption needs Seb's buy-in** — this is a foundational change to HIS codebase. The document makes the case but the conversation hasn't happened.
- **MemPalace KG empty** (0 entities, 0 triples) — decision pending on whether to populate
- **ARC + chamber-library pushes** still blocked on Gitea
- **External Auditor response** pending

## Next actions

1. **Discuss systemic fixes with Seb** — bring the consolidated document. Effect-TS adoption is the biggest conversation.
2. **Respond to External Auditor** — assessment pending, what enters record, whether HOLD triggered
3. **Answer DN-RET-02 practitioner questions** — score memorization, crowding, re-learning
4. **Push ARC + chamber-library** when Gitea is resolved
5. **Continue /audit of vault inbox** — 16 files remain

## Commits this session

- `487503e` — research folder, recall path audit, MemPalace + Hermes Agent studies
- `d1d4b16` — same (rebased after pull)
- `3088868` — L1 pipeline landscape
- `c35f621` — L1 full pipeline audit
- `ce60b4c` — L1 systemic fixes roadmap

## Issues filed this session

| # | Title | Severity |
|---|-------|----------|
| #124 | Ollama silent recall failure | HIGH |
| #125 | BM25 scoring broken | MEDIUM |
| #126 | Normalization aggressiveness | LOW |
| #133 | Module readiness lie | CRITICAL |
| #134 | Entity error swallowing | HIGH |
| #135 | Temporal query silent failure | HIGH |
| #136 | Vector battery skip | HIGH |

## No Thistleweld observations this session