[FIX] L1 pin root-caused and cleared; S-series closed; instrument census

mindfabric-00 had been event-loop-pinned for 6+ days (100% CPU, /health silent).
Profile + CDP inspector named two hot paths, both from runTemporalPipeline:

  checkForCycle -> getCausalEdgesFromSqlite   99.8% of samples
  tryExtendChains -> getChainsContainingSeq   now dominant (json_each scan)

Cause of the first: ANALYZE had never been run, so SQLite preferred a boolean
index (idx_caused_tombstoned, matching ~all 836k edges) over idx_caused_from.
ANALYZE across 15 module DBs flipped the plan; 6.4x on a microbenchmark and
99.8% -> 6.0% in the live profile. /health went from silent to 200 in 0.13s.

B1.1's fan-out cap is IMPLEMENTED AND WORKING (today: max in-degree exactly 20,
zero violations; pre-23-June: max 629, avg 67.6). The defect is data, not code —
836k edges / 813k chains minted under ungoverned fan-out before the fix landed.
Repair run: derived stores wiped, logchain preserved, replay in flight.

S-series closed (jurist had already ruled all of Q1-Q5 on 2026-05-18):
  S6/S7/S9 implemented (Symmetria §3 flags, `suspend` outcome, wrap-up §8 tenses)
  S2 rebuilt as [FIX] — wake-digest unwrapped-session detector, discrimination-
    gated on real sessions (11 wrapped / 2 unwrapped)
  S4/S5 withdrawn with MemPalace (steward ruling)
Dormant legacy dispositioned: PENDING-4/5/11/12, CD-03, ICP-19 duplicate.
Open authorization items 22 -> 10.

Census 01: which instruments have no real negative instance. Finding — the
governance drift-check has 3 of 5 families inert against the current CLAUDE.md,
and 71 of 75 verification-ladder entries are cited nowhere outside the ladder.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuMjg3ipEVa3n8CoSzoyvc
This commit is contained in:
David F Glidden
2026-08-03 20:57:41 +02:00
co-authored by Claude Opus 5
parent 23e7515302
commit 0a48e6934d
8 changed files with 427 additions and 16 deletions
+25 -1
View File
@@ -18,7 +18,23 @@ permalink: claude-memory/project-l1-reliability
---
## Current state (as of 2026-06-06)
## Current state (as of 2026-08-03)
- **The baton rule is LIFTED.** The 2026-06-06 instruction — *"Do not re-enter L1 until Seb responds or pushes — his move"* — was steward-lifted 2026-08-03 (*"if we can move this ahead then let's do what it takes"*). Context the steward supplied: Seb is not silent by choice; he is deep in a frustrating rabbit hole on Peter's side project, which borrows heavily from L1. Seb's last commit remains **2026-06-07**; the steward's own **2026-06-23** (`#175` N6, `/health` auth tiering).
- **mindfabric-00 was event-loop-pinned for 6+ days and nobody was looking.** 100% CPU, `/health` silent while port 3011 listened, PID 874 up 6d 9h. Same signature as the June pin, *after* the N6 fix.
- **Pin root-caused by profile, then by inspector — two hot paths, both June-named.** Main thread: `uv__run_timers` → `Statement::JS_all` (better_sqlite3 **synchronous** `.all()`) → SQLite btree traversal. CDP profile named them exactly:
1. `runTemporalPipeline` → `checkForCycle` → **`getCausalEdgesFromSqlite`** (`storage-sqlite.ts:359`) — **99.8% of samples. FIXED (see below) → 6.0%.**
2. `runTemporalPipeline` → `tryExtendChains` → **`getChainsContainingSeq`** (`storage-sqlite.ts:657`) — now dominant at **88%**. `SELECT * FROM causal_chain WHERE EXISTS (SELECT 1 FROM json_each(event_seqs) WHERE value = ?)` — a full scan of 813,178 chains, **measured 4.0 s per call**. **Unindexable as written**; needs `event_seqs` normalised into an indexed join table. Schema change ⇒ Seb's call. STILL OPEN.
- **`ANALYZE` had never been run in four months.** With no statistics, SQLite preferred `idx_caused_tombstoned` (a **boolean** index matching ~all 836k rows) over `idx_caused_from`. Every cycle-check query therefore scanned the whole edge table. `ANALYZE` on all 15 module DBs (4.4 s total) flipped the plan to `idx_caused_from`; microbenchmark **3.671 s → 0.573 s (6.4×)**; production profile 99.8% → 6.0%.
- **B1.1 fan-out cap is IMPLEMENTED AND WORKING — the code is not the problem.** `causal_edges_per_event_cap: 20` (`temporal/types.ts:201`), applied at `pipeline.ts:276`. Edges created **today: max in-degree exactly 20, avg 13.4, zero violations.** Edges created **pre-2026-06-23: max 629, avg 67.6, 4,677 nodes over cap.**
- **Therefore the defect is DATA, not CODE.** 836,467 edges / 813,178 chains were minted 2026-04-18 → 2026-06-23 under ungoverned fan-out. The fix landed *after* the damage, and the legacy graph taxes every future operation because `getChainsContainingSeq` scans all chains regardless of how disciplined new writes are. **This is why no ingest ever completed: cost per event grew with corpus size — divergent, not merely slow.** It also explains #65's "each attempt fails differently" — same wall, different points on the curve.
- **`coherence_evaluated = 0` of 813,178 chains — 0.00%, not one, ever.** June recorded "97%+ unevaluated"; it is now categorically zero. The epistemic layer that would prune chains has never executed. Separate from the pin; unfixed by the repair (a rebuild resets the count, it does not start the evaluator).
- **REPAIR RUN 2026-08-03 (steward-authorized).** `cm-ctl.sh repair --yes` — wiped `sqlite/ lance/ reasonchain/ perception/`, **preserved `logchain/` (145 MB, 109 files)**. Backup `mindfabric-00-20260803-wedged-preinvestigation` (1.4 G, 256/256 files, `quick_check: ok` ×4) taken **before** any change. Replay in flight, fan-out governed from the first event (`max_in=2, over_cap20=0`), **~10 nodes/min → ~33 h** for ~19,925 events. Finite now where it was divergent. Survives session end (launchd `com.capablemind.bettermemories`). **⚠ No `KeepAlive`** — a crash leaves it down silently. Steward has caffeine on indefinitely.
- **Other findings, unfixed:** **4,648 empty Lance index directories** under `vector_chunks.lance/_indices/` (4,653 dirs, 5 real), leaking ~40/day since April. **#176 confirmed in source** — `BackupOrchestrator.countUnprotectedEntries` returns `0` when `lastBackupSuccess` is null, i.e. never-backed-up reads as fully protected.
- **GH backlog swept** per Seb's own standing instructions (`#170`, filed 2026-04-23, **never delivered** — circle-forward push failed, GitHub fallback unread for 3½ months). Closed `#125` (superseded: vector storage moved to LanceDB) and `#151` (done by `4f5b870`); narrowed `#121` to its live second cause (latency baseline frozen after first N samples, `resource-monitor.ts:145`).
- **Pattern across the day:** governors exist and do not engage — coherence evaluation never run, `ANALYZE` never run, `countUnprotectedEntries` safe-cases an absence, drift-check 3-of-5 families inert, verification ladder 71-of-75 entries never cited.
## Previous state (as of 2026-06-06 — superseded 2026-08-03)
- **The reply arc is COMPLETE and the baton is with Seb.** Three notes pushed 2026-06-06 (CM-AI `b255e74` → `0ecb45e` → `2e87bab`): Tier A reply to his cover note (`b4daa71`) + same-day addendum + Tier B code-grounded read. Do not re-enter L1 until Seb responds or pushes — his move.
- **mindfabric-00 root-caused**: found event-loop-pinned 4 days; CPU-profiled (proven, not inferred) to the temporal causal-chain path — `getChainsContainingSeq` `json_each` full scan over 272,172 chains per created link + chain-bloat feedback (new 2-node chain per no-extension) + uncancelled 30s timeout. Restarted clean; **re-wedge expected** as temporal re-encounters the dense pre-B1.1 graph (280k edges / 7,959 nodes, top fan-out 324); restart = holding pattern. Backup at `~/.capablemind/backups/mindfabric-00-20260606-prefix-state` (same-disk APFS clone, 224/224 verified).
@@ -172,6 +188,14 @@ Each verified against `BetterMemories.io@3bc8b75` source. Re-verification agains
## Chronological log (most recent first; append substantive moves)
### 2026-08-03 — The wedge named, ANALYZE applied, rebuild in flight; the code was right and the data was not
- **Entered L1 against the standing baton rule, on explicit steward lift.** The session had first proposed a *clasp* build (M4 Pro as circle-inference clasp per 43L/43M) to unblock ingest — measured and abandoned: §4.13's 2000 ms budget vs 192 ms transatlantic RTT, and a benchmark of Qwen3.6-35B-A3B-8bit on the M4 (median 2927 ms for 150 tokens, ~92-token break-even). **The clasp was the wrong lever** — the instance was never inference-bound. Profiling the running process answered in five minutes what reasoning from specs had not.
- **Method that worked, and is the transferable lesson:** ask the machine, not the specs. `sample` → main thread pinned in a timer callback; SIGUSR1 → CDP inspector → exact JS frames; read-only row counts and `EXPLAIN QUERY PLAN` off a *backup copy*, never the live DB.
- Diagnosis, fix, verification and the repair: see **Current state** above. Health went from silent-for-6-days to **200 in 0.13 s**.
- **Corrections made in-session, recorded because they were load-bearing:** (a) hypothesised the `json_each` scan was the pin — it was the *second* path, masked by the cycle-check; (b) hypothesised `slice(0, undefined)` had made B1.1 inert — **wrong, the cap works perfectly**; (c) reported "never backed up" — false, four manual clones exist; `backup.last_success: null` refers to the *automated* pipeline only; (d) reported PENDING-82 not installed — wrong config surface, it has been installed in the Claude.app desktop config all along.
- **Steward framing to carry:** the distance from being away is what made this visible. *"Unless the time off gives us the necessary distance to identify what we couldn't see by being in too intensively."* Nothing found today needed cleverness; it needed someone to profile a running process and count rows in a table — which nobody does from inside the work, because from inside you reason about the code you just wrote.
### 2026-06-07 — Ops finding: training-loop Ollama client leaks connections, retries a hard-down dependency forever (A-class; note-don't-send)
- **Trigger:** steward asked "why is ollama devouring my cpu?" Diagnosis: a two-failure pile-up. (1) The morning's brew upgrade to ollama 0.30.6 installed a **bottle with no `llama-server` runner** (known upstream: ollama#16535, homebrew-core#285917) — every model load 500'd after paying full setup cost. (2) **mindfabric-00 held 829 open TCP connections to Ollama** and re-sent `/api/chat` endlessly — requests 18s–2m47s in scheduler retry loops, ~50 POSTs per log window, for hours. No backoff, no circuit breaker, connections never released. Ollama serve at ~76% CPU with zero models loaded.
- **The finding for Seb (joins the uncancelled-timeout family from the 06-06 CPU profile):** the training/graduation loop's Ollama client treats a hard-down dependency as transient indefinitely. A2.4 (degraded-environment refusal) is the constitutional shape; mechanically it wants: bounded retries with backoff, a circuit breaker on consecutive 5xx, and connection release on request failure. **Not sent — baton with Seb; relay at next exchange.**
@@ -0,0 +1,73 @@
---
name: Session Ledger 2026-08-03
description: Practice-of-return ledger maintained by /symmetria — returns, open horizons, recalibrations, authorization moves, sub-agent dialogues, bypasses.
type: feedback
---
# Session Ledger — 2026-08-03
## Returns
- **09:37 — named at wake, before any action.** The inherited pulling thread (CONTROL-A v2) was parked on a stated reason: *distance from the hand that concealed a defect inside that document*. The pause is 41 minutes. Naming this rather than opening the document, because a hold whose reason has not been satisfied is not discharged by the calendar rolling over. Surfaced to the steward as a judgment for them, not a decision I make.
- **10:1x — prediction 2 fell, and the pre-registration made that usable.** I predicted `test_reduce.py`/`test_twin.py` would be fixture-based and silent about it. Both are property-derived and explicit. Because the prediction was on record, the miss became evidence about *where* decay lives (old instruments, not new) rather than something I could quietly not mention.
- **10:2x — declined to bank the happy result.** When the `fool/` population came out mostly sound, the pre-registration's own clause required testing "did I go lenient?" before accepting it. Testing it moved the census to the wake instruments, which is where the real finding was.
## What held
- Wake substrate-check fired correctly: `fool/` is at `~/dotfiles/claude/governance/fool/`, not the path I first guessed. Every artifact the wrap named is present (both kernels, both controls + tags + units, `twin-ledger.json` with `inherited_defects`, four test suites, three pre-registrations). Nothing half-edited.
- The 22 open authorization items were reported to the steward as **unverified against substrate**, per the wake's disposition-clause-is-not-a-status rule, rather than stated flat.
## Open horizons
- **CONTROL-A v2** — four defects, not two (jurist Findings 1 & 2 + candidates I2, I3). I3 attacks the conclusion at its root. Held on distance.
- **The literal question** — census of every gate in `fool/` and every verification-ladder entry: does it name the real artifact it was shown failing on, or does its absence look like coverage? Answerable now; needs no distance.
- **False-positive rate: still unmeasured after four attempts.** Prediction on record from the last wrap: a *fifth new reason* would stop being reassuring.
- Correlation 02 (formation-same pair: jurist vs executor) → PENDING-89.
- Skill-harvest register exceeds read caps; compaction owed.
## Findings — census 01 (full result in `governance/fool/census-01-negative-instances-RESULT.md`)
- **`governance-drift-check.py`: 3 of 5 check families are INERT against the current `~/CLAUDE.md`** (MCP-tools, hooks, horizons — 0 matches each). All controls pass; the hooks control sits inside its own `if` and does not run. The morning "0 substrate-contradicted claims" is honest about what it measured and silent about what it can no longer see. **[HARDENING] — owed to PENDING.**
- **New class named:** *a drift-checker keyed to a document's current wording goes inert when the document is improved.* Family 2 died because of the obligation-before-instrument rewrite — the rule that protects doctrine from instrument-decay silently retired its own watchman.
- **Path family silently skips any backticked path containing a space** — the Compass vault path is never checked. Exists today; a break would still report 0.
- **71 of 75 verification-ladder entries are cited by name nowhere outside the ladder** (grep positive-controlled first). Does NOT establish they never fire — establishes the record cannot distinguish firing from silence. Root cause: **ladder entries have no IDs**, so firing history is unrecordable by construction.
- **`~/PENDING.md`: PENDING-4 self-reports "Implemented and verified" and is counted OPEN** (digest recognises `(CLOSED)`/`COMPLETED` headers, not a body `**Status:**`). **ICP-19 Remit Expansion is recorded twice** (L74, L93 — same date opened, same requirement): a one-canonical-source violation.
## Confidence to recalibrate
- Drift to hold today: **removing a claim conceals the reliance** (earned yesterday, in my own hand).
- What compounded: **pre-registration written before the run** — stopped trial 04 repeating trial 03's fatal omission.
## Authorization moves
- **S-series closed, 22 → 16 open.** Steward delegated the choice (*"Which would you choose? This is tooling for you…"*) and ruled MemPalace-associated functionality retired. Acted on that as authorization for the three items already awaiting only a steward nod, and as a disposition for the two MemPalace-dependent ones.
- **S6 IMPLEMENTED** — Symmetria §3, six flags.
- **S7 IMPLEMENTED** — Symmetria §6, `suspend` outcome.
- **S9 IMPLEMENTED** — wrap-up §8, three tenses (Past and Present were wholly absent; Future was already complete).
- **S2 CLOSED, rebuilt as [FIX]** — `wake-digest.py` unwrapped-session detector, positive-controlled + discrimination-gated on real sessions (11 wrapped / 2 unwrapped).
- **S4, S5 WITHDRAWN** — retired with MemPalace. S4's concept harvested into the §3 flag; S5's race provably cannot recur.
- **Framing challenged rather than executed.** The steward asked me to *"prepare everything for the jurist."* The jurist had already ruled on all of Q1–Q5 on 2026-05-18 and had explicitly assigned the one open piece to the steward. Building the requested package would have spent the jurist on answered questions and dead mechanisms. Said so before building.
- **Not acted on, flagged for the steward:** PENDING-82 is installed (verified in the Claude.app desktop config), which also bears on PENDING-86. Both still counted open. Disposition is the steward's.
### Dormant legacy against the repos — 16 → 10 open
- **`betterMemories_app#170` is a standing instruction addressed to David's Claude Code, undelivered since 2026-04-23.** Its own body records why: circle-forward delivery failed (*"2 of 4 paired peers failed delivery with `fetch failed` — David's BMF was offline"*), and the GitHub copy was the durable fallback nobody read. It prescribes exactly this task (periodic backlog review; close/narrow/leave/supersede; verify before closing on someone else's evidence). **Three months of a channel failing open.**
- **PENDING-4 CLOSED** — verified against substrate, not from its own `Status: Implemented and verified` line. GH #120 and #133 closed; four corroborating commits.
- **PENDING-5 CLOSED** — resolved via GH #124/#135 + `bd1f650`. ⚠ **Class still open at #165** (H2 silent recall failure on battery, priority:critical) — closing the instance must not read as closing the class.
- **PENDING-11 / PENDING-12 CLOSED** — both ruled AUTHORIZED on **2026-03-23**. Two independent defects kept them open 4½ months: the REVIEWED blocks are **indented one space** (3 such orphans in the whole file, fence-tracked census), *and* their headers name no PENDING, so resolution-by-name finds nothing. `REVIEWED.md` is not mine to correct — flagged.
- **CD-03 CLOSED as a register entry** — its own status is OPERATIVE; it was a notification counted as an open constitutional decision for four months. Declaration untouched.
- **ICP-19 CLOSED as duplicate** — two records, one obligation, both dated 2026-04-07.
- **PENDING-10 is NOT dormant** — my classification was wrong; its 2026-08-02 amendment scopes it into the open #176 backup-pipeline audit.
- **Root cause, both halves:** the register and GitHub diverged. L1 work migrated to issues; `PENDING.md` kept March snapshots and no one reconciled. The governance half had the mirror problem — rulings existed, tooling couldn't see them.
- **Flagged, not fixed (Seb's territory):** `BetterMemories.io/CLAUDE.md` claims "embedded SurrealDB", says tests use in-memory SurrealDB, and lists `src/core/storage/surrealdb.ts` — which is absent. Dependency is `better-sqlite3`; **steward confirmed SurrealDB is legacy**. This also supersedes the premise of open issue **#125** (BM25 broken in SurrealDB 3.0.1).
### Correction
**Instrument error, self-caught mid-task.** I told the steward "80 of 158 REVIEWED headers are indented" — a ~27× overstatement produced by `^\s+## REVIEWED`, where `\s` matched the preceding newline and so counted every flush-left header. The fence-tracked census gives **3**. Third time in one day that a dramatic number from an uncontrolled instrument had to be walked back; the census I wrote this morning names exactly this failure and I still shipped the number before checking it.
Earlier this session I reported PENDING-82 as "built, not installed", having checked **Claude Code's** MCP configs — the wrong surface for a jurist-facing server. Steward corrected. The right file (`~/Library/Application Support/Claude/claude_desktop_config.json`) has had it since it was built.
## Sub-agent dialogues
## Bypasses