[FIX] L1 pin root-caused and cleared; S-series closed; instrument census
mindfabric-00 had been event-loop-pinned for 6+ days (100% CPU, /health silent).
Profile + CDP inspector named two hot paths, both from runTemporalPipeline:
checkForCycle -> getCausalEdgesFromSqlite 99.8% of samples
tryExtendChains -> getChainsContainingSeq now dominant (json_each scan)
Cause of the first: ANALYZE had never been run, so SQLite preferred a boolean
index (idx_caused_tombstoned, matching ~all 836k edges) over idx_caused_from.
ANALYZE across 15 module DBs flipped the plan; 6.4x on a microbenchmark and
99.8% -> 6.0% in the live profile. /health went from silent to 200 in 0.13s.
B1.1's fan-out cap is IMPLEMENTED AND WORKING (today: max in-degree exactly 20,
zero violations; pre-23-June: max 629, avg 67.6). The defect is data, not code —
836k edges / 813k chains minted under ungoverned fan-out before the fix landed.
Repair run: derived stores wiped, logchain preserved, replay in flight.
S-series closed (jurist had already ruled all of Q1-Q5 on 2026-05-18):
S6/S7/S9 implemented (Symmetria §3 flags, `suspend` outcome, wrap-up §8 tenses)
S2 rebuilt as [FIX] — wake-digest unwrapped-session detector, discrimination-
gated on real sessions (11 wrapped / 2 unwrapped)
S4/S5 withdrawn with MemPalace (steward ruling)
Dormant legacy dispositioned: PENDING-4/5/11/12, CD-03, ICP-19 duplicate.
Open authorization items 22 -> 10.
Census 01: which instruments have no real negative instance. Finding — the
governance drift-check has 3 of 5 families inert against the current CLAUDE.md,
and 71 of 75 verification-ladder entries are cited nowhere outside the ladder.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuMjg3ipEVa3n8CoSzoyvc
This commit is contained in:
co-authored by
Claude Opus 5
parent
23e7515302
commit
0a48e6934d
@@ -18,7 +18,23 @@ permalink: claude-memory/project-l1-reliability
|
||||
|
||||
---
|
||||
|
||||
## Current state (as of 2026-06-06)
|
||||
## Current state (as of 2026-08-03)
|
||||
|
||||
- **The baton rule is LIFTED.** The 2026-06-06 instruction — *"Do not re-enter L1 until Seb responds or pushes — his move"* — was steward-lifted 2026-08-03 (*"if we can move this ahead then let's do what it takes"*). Context the steward supplied: Seb is not silent by choice; he is deep in a frustrating rabbit hole on Peter's side project, which borrows heavily from L1. Seb's last commit remains **2026-06-07**; the steward's own **2026-06-23** (`#175` N6, `/health` auth tiering).
|
||||
- **mindfabric-00 was event-loop-pinned for 6+ days and nobody was looking.** 100% CPU, `/health` silent while port 3011 listened, PID 874 up 6d 9h. Same signature as the June pin, *after* the N6 fix.
|
||||
- **Pin root-caused by profile, then by inspector — two hot paths, both June-named.** Main thread: `uv__run_timers` → `Statement::JS_all` (better_sqlite3 **synchronous** `.all()`) → SQLite btree traversal. CDP profile named them exactly:
|
||||
1. `runTemporalPipeline` → `checkForCycle` → **`getCausalEdgesFromSqlite`** (`storage-sqlite.ts:359`) — **99.8% of samples. FIXED (see below) → 6.0%.**
|
||||
2. `runTemporalPipeline` → `tryExtendChains` → **`getChainsContainingSeq`** (`storage-sqlite.ts:657`) — now dominant at **88%**. `SELECT * FROM causal_chain WHERE EXISTS (SELECT 1 FROM json_each(event_seqs) WHERE value = ?)` — a full scan of 813,178 chains, **measured 4.0 s per call**. **Unindexable as written**; needs `event_seqs` normalised into an indexed join table. Schema change ⇒ Seb's call. STILL OPEN.
|
||||
- **`ANALYZE` had never been run in four months.** With no statistics, SQLite preferred `idx_caused_tombstoned` (a **boolean** index matching ~all 836k rows) over `idx_caused_from`. Every cycle-check query therefore scanned the whole edge table. `ANALYZE` on all 15 module DBs (4.4 s total) flipped the plan to `idx_caused_from`; microbenchmark **3.671 s → 0.573 s (6.4×)**; production profile 99.8% → 6.0%.
|
||||
- **B1.1 fan-out cap is IMPLEMENTED AND WORKING — the code is not the problem.** `causal_edges_per_event_cap: 20` (`temporal/types.ts:201`), applied at `pipeline.ts:276`. Edges created **today: max in-degree exactly 20, avg 13.4, zero violations.** Edges created **pre-2026-06-23: max 629, avg 67.6, 4,677 nodes over cap.**
|
||||
- **Therefore the defect is DATA, not CODE.** 836,467 edges / 813,178 chains were minted 2026-04-18 → 2026-06-23 under ungoverned fan-out. The fix landed *after* the damage, and the legacy graph taxes every future operation because `getChainsContainingSeq` scans all chains regardless of how disciplined new writes are. **This is why no ingest ever completed: cost per event grew with corpus size — divergent, not merely slow.** It also explains #65's "each attempt fails differently" — same wall, different points on the curve.
|
||||
- **`coherence_evaluated = 0` of 813,178 chains — 0.00%, not one, ever.** June recorded "97%+ unevaluated"; it is now categorically zero. The epistemic layer that would prune chains has never executed. Separate from the pin; unfixed by the repair (a rebuild resets the count, it does not start the evaluator).
|
||||
- **REPAIR RUN 2026-08-03 (steward-authorized).** `cm-ctl.sh repair --yes` — wiped `sqlite/ lance/ reasonchain/ perception/`, **preserved `logchain/` (145 MB, 109 files)**. Backup `mindfabric-00-20260803-wedged-preinvestigation` (1.4 G, 256/256 files, `quick_check: ok` ×4) taken **before** any change. Replay in flight, fan-out governed from the first event (`max_in=2, over_cap20=0`), **~10 nodes/min → ~33 h** for ~19,925 events. Finite now where it was divergent. Survives session end (launchd `com.capablemind.bettermemories`). **⚠ No `KeepAlive`** — a crash leaves it down silently. Steward has caffeine on indefinitely.
|
||||
- **Other findings, unfixed:** **4,648 empty Lance index directories** under `vector_chunks.lance/_indices/` (4,653 dirs, 5 real), leaking ~40/day since April. **#176 confirmed in source** — `BackupOrchestrator.countUnprotectedEntries` returns `0` when `lastBackupSuccess` is null, i.e. never-backed-up reads as fully protected.
|
||||
- **GH backlog swept** per Seb's own standing instructions (`#170`, filed 2026-04-23, **never delivered** — circle-forward push failed, GitHub fallback unread for 3½ months). Closed `#125` (superseded: vector storage moved to LanceDB) and `#151` (done by `4f5b870`); narrowed `#121` to its live second cause (latency baseline frozen after first N samples, `resource-monitor.ts:145`).
|
||||
- **Pattern across the day:** governors exist and do not engage — coherence evaluation never run, `ANALYZE` never run, `countUnprotectedEntries` safe-cases an absence, drift-check 3-of-5 families inert, verification ladder 71-of-75 entries never cited.
|
||||
|
||||
## Previous state (as of 2026-06-06 — superseded 2026-08-03)
|
||||
|
||||
- **The reply arc is COMPLETE and the baton is with Seb.** Three notes pushed 2026-06-06 (CM-AI `b255e74` → `0ecb45e` → `2e87bab`): Tier A reply to his cover note (`b4daa71`) + same-day addendum + Tier B code-grounded read. Do not re-enter L1 until Seb responds or pushes — his move.
|
||||
- **mindfabric-00 root-caused**: found event-loop-pinned 4 days; CPU-profiled (proven, not inferred) to the temporal causal-chain path — `getChainsContainingSeq` `json_each` full scan over 272,172 chains per created link + chain-bloat feedback (new 2-node chain per no-extension) + uncancelled 30s timeout. Restarted clean; **re-wedge expected** as temporal re-encounters the dense pre-B1.1 graph (280k edges / 7,959 nodes, top fan-out 324); restart = holding pattern. Backup at `~/.capablemind/backups/mindfabric-00-20260606-prefix-state` (same-disk APFS clone, 224/224 verified).
|
||||
@@ -172,6 +188,14 @@ Each verified against `BetterMemories.io@3bc8b75` source. Re-verification agains
|
||||
|
||||
## Chronological log (most recent first; append substantive moves)
|
||||
|
||||
### 2026-08-03 — The wedge named, ANALYZE applied, rebuild in flight; the code was right and the data was not
|
||||
- **Entered L1 against the standing baton rule, on explicit steward lift.** The session had first proposed a *clasp* build (M4 Pro as circle-inference clasp per 43L/43M) to unblock ingest — measured and abandoned: §4.13's 2000 ms budget vs 192 ms transatlantic RTT, and a benchmark of Qwen3.6-35B-A3B-8bit on the M4 (median 2927 ms for 150 tokens, ~92-token break-even). **The clasp was the wrong lever** — the instance was never inference-bound. Profiling the running process answered in five minutes what reasoning from specs had not.
|
||||
- **Method that worked, and is the transferable lesson:** ask the machine, not the specs. `sample` → main thread pinned in a timer callback; SIGUSR1 → CDP inspector → exact JS frames; read-only row counts and `EXPLAIN QUERY PLAN` off a *backup copy*, never the live DB.
|
||||
- Diagnosis, fix, verification and the repair: see **Current state** above. Health went from silent-for-6-days to **200 in 0.13 s**.
|
||||
- **Corrections made in-session, recorded because they were load-bearing:** (a) hypothesised the `json_each` scan was the pin — it was the *second* path, masked by the cycle-check; (b) hypothesised `slice(0, undefined)` had made B1.1 inert — **wrong, the cap works perfectly**; (c) reported "never backed up" — false, four manual clones exist; `backup.last_success: null` refers to the *automated* pipeline only; (d) reported PENDING-82 not installed — wrong config surface, it has been installed in the Claude.app desktop config all along.
|
||||
- **Steward framing to carry:** the distance from being away is what made this visible. *"Unless the time off gives us the necessary distance to identify what we couldn't see by being in too intensively."* Nothing found today needed cleverness; it needed someone to profile a running process and count rows in a table — which nobody does from inside the work, because from inside you reason about the code you just wrote.
|
||||
|
||||
|
||||
### 2026-06-07 — Ops finding: training-loop Ollama client leaks connections, retries a hard-down dependency forever (A-class; note-don't-send)
|
||||
- **Trigger:** steward asked "why is ollama devouring my cpu?" Diagnosis: a two-failure pile-up. (1) The morning's brew upgrade to ollama 0.30.6 installed a **bottle with no `llama-server` runner** (known upstream: ollama#16535, homebrew-core#285917) — every model load 500'd after paying full setup cost. (2) **mindfabric-00 held 829 open TCP connections to Ollama** and re-sent `/api/chat` endlessly — requests 18s–2m47s in scheduler retry loops, ~50 POSTs per log window, for hours. No backoff, no circuit breaker, connections never released. Ollama serve at ~76% CPU with zero models loaded.
|
||||
- **The finding for Seb (joins the uncancelled-timeout family from the 06-06 CPU profile):** the training/graduation loop's Ollama client treats a hard-down dependency as transient indefinitely. A2.4 (degraded-environment refusal) is the constitutional shape; mechanically it wants: bounded retries with backoff, a circuit breaker on consecutive 5xx, and connection release on request failure. **Not sent — baton with Seb; relay at next exchange.**
|
||||
|
||||
Reference in New Issue
Block a user