session 2026-08-04: PENDING-92/93/94 + the replay-never-resumed finding + l1-replay-sampler
Filed PENDING-92 [HARDENING] idle ladder (cool/deep unreachable, spec §9A.1 divergence), PENDING-93 [PROPOSAL] event_seqs normalisation, PENDING-94 [ESCALATE] the resume floor — minCursor pinned at 0 by two non-participating modules, so 13/13 restarts rebuilt from seq 0 and the catch-up branch has never executed. Recall never worked either (retrieval_count = 0 across the whole April-June graph); same fact from the other end. Adds scripts/l1-replay-sampler.py (external read-only sampler, four positive controls, refuses to run blind). Note to Seb pushed separately as CapableMind-AI@ad285df. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuMjg3ipEVa3n8CoSzoyvc
This commit is contained in:
co-authored by
Claude Opus 5
parent
ad2fb07698
commit
9c2a7c985c
+48
@@ -754,3 +754,51 @@ Measured against the register itself before acting. Four corrections; the item's
|
|||||||
**Verification:** 33/33 quotations in the package mechanically contained against source, 9/9 positive controls absent, instrument verified. Prototype checks: zero-JS gate 0, cycle-end clamp holds at three horizons and floors on a negative trajectory, monotonic over 800 sampled points, validator rejects 6/6 malformed genomes by name.
|
**Verification:** 33/33 quotations in the package mechanically contained against source, 9/9 positive controls absent, instrument verified. Prototype checks: zero-JS gate 0, cycle-end clamp holds at three horizons and floors on a negative trajectory, monotonic over 800 sampled points, validator rejects 6/6 malformed genomes by name.
|
||||||
**Files affected:** `docs/AldineXXI-Codex/drafts/vignette-phase-1a-JURIST-PACKAGE-2026-08-02.md` (new); `tools/vignette-proto/` (built, committed `946b88b`/`1d40d4d`/`095be00`). No spec text changed, nothing integrated.
|
**Files affected:** `docs/AldineXXI-Codex/drafts/vignette-phase-1a-JURIST-PACKAGE-2026-08-02.md` (new); `tools/vignette-proto/` (built, committed `946b88b`/`1d40d4d`/`095be00`). No spec text changed, nothing integrated.
|
||||||
**Awaiting:** Steward relay to the jurist, and a decision on Q5 before the ruling is sought.
|
**Awaiting:** Steward relay to the jurist, and a decision on Q5 before the ruling is sought.
|
||||||
|
|
||||||
|
## PENDING-92 — The idle ladder's bottom half is unreachable, and the work that lives there has never run
|
||||||
|
**Date:** 2026-08-04
|
||||||
|
**Tag:** [HARDENING]
|
||||||
|
**Summary:** `cool` and `deep` are unreachable on this deployment — the only `warm → cool` path is `onAgentDisconnect()`, whose sole caller is the shutdown sequence — so deferrable `idle_only` work (background training, consolidation, Ollama batch windows) has never drained once.
|
||||||
|
**Rationale:** Sixth instance of the class named 2026-08-03 (governor exists and never engages), and the first found by looking for it rather than tripping over it. An inert control reports success: `bm_idle_state` has only ever reported 0 or 1, which reads as a healthy machine rather than a ladder missing its bottom half.
|
||||||
|
**Evidence (substrate, positive-controlled):** across the full `bmf.stderr.log`, 2,393 idle transitions in exactly two shapes — `active → warm` (1,197) and `warm → active` (1,196). **Zero** to `cool` or `deep`, ever. **Zero** `drained N deferrable idle_only work items` lines, ever. The positive control is the 2,393 itself: the grep demonstrably sees transitions when they exist, so the absence is measured, not assumed.
|
||||||
|
**Code:** `src/inference/idle-state-machine.ts:204` — *"warm and deep have no time-based exits from tick"*; `tick()` handles only `active` and `cool`. The single `onAgentDisconnect()` caller is `src/bootstrap.ts:2460`, inside the shutdown path (*"1. Stop MCP server and signal agent disconnect (44G)"*). `cool` is therefore reachable only while the process is dying, and `deep` — which requires 4h resident in `cool` — is unreachable by construction.
|
||||||
|
**Spec (the divergence):** `docs/specs/operations/local-inference-spec.md` §9A.1 (v1.6, Amendment 44G), HTTP-only/service-mode table: *"State machine transitions are identical — only the detection inputs change. 'Agent connected' maps to 'recent HTTP activity.' 'Agent disconnected' maps to 'no HTTP activity for cool idle threshold.'"* The spec requires a time-based path into `cool`; the code implements only literal MCP disconnect. The same section anticipates precisely this deployment: *"For idle to work beyond the session, the architecture needs launchd/systemd as the primary process with stdio MCP connecting TO the service."* That is mindfabric-00 exactly.
|
||||||
|
**Why [HARDENING] and not [FIX], though the divergence is scoped:** the *diagnosis* is FIX-shaped — code diverges from spec, and per the repo's methodology the spec wins. The *remedy* is not: enabling the descent switches on a subsystem that has never executed in production on any instance. Idle training cycles, memory consolidation and Ollama batch windows would run for the first time, on the steward's primary instance, against a graph mid-rebuild. Calling that "restoring intended behaviour" would be a self-assessment I cannot honestly make.
|
||||||
|
**Options:** (a) implement the spec's HTTP-activity-timeout path behind an env flag defaulting **off**, so the first descent is deliberate and observed; (b) implement unflagged, matching spec directly; (c) amend the spec instead — declare `cool`/`deep` out of scope for MCP-attached deployments and delete the unreachable states, choosing honest degradation over dormant capability; (d) defer entirely until the replay completes.
|
||||||
|
**Recommendation:** (a) sequenced behind (d) — build now on a branch, enable only after the replay completes and with `l1-replay-sampler.py` running, so the first-ever `warm → cool → deep` descent is watched rather than discovered afterwards. (c) stays live as the honest alternative if the idle subsystem turns out to be unwanted on laptops; what should not persist is a four-state ladder advertising two states it cannot enter.
|
||||||
|
**Files affected:** `src/inference/idle-state-machine.ts`, `src/bootstrap.ts` (BetterMemories.io). Spec unchanged — the code moves to the spec, not the reverse.
|
||||||
|
**Awaiting:** Steward authorization; then Seb review via PR per Constitutional Constraint #3 (no direct push to main on L1).
|
||||||
|
|
||||||
|
## PENDING-93 — `getChainsContainingSeq`: the rebuild buys a constant factor, not a complexity class
|
||||||
|
**Date:** 2026-08-04
|
||||||
|
**Tag:** [PROPOSAL]
|
||||||
|
**Summary:** `SELECT * FROM causal_chain WHERE EXISTS (SELECT 1 FROM json_each(event_seqs) WHERE value = ?)` is a full scan of every chain, unindexable as written; the 2026-08-03 repair reduces how many chains it scans but leaves the cost linear in chain count.
|
||||||
|
**Rationale:** This is the second of the two hot paths named by the 2026-08-03 CDP profile (the first, `getCausalEdgesFromSqlite`, was resolved by `ANALYZE`). It is the one that remains, and the repair does not address it. Filing it so the schema question is docketed rather than living only in the workstream tracker — PENDING-11/12 sat open four and a half months precisely because they were recorded where nobody re-read them.
|
||||||
|
**Measured:** 4.0 s per call against 813,178 chains (2026-08-03, off the verified backup). The current rebuild at 22,300/39,089 events (57%) already carries **71,225 chains against 6,803 nodes — 10.47 chains per node**. A linear extrapolation to full replay gives ~125,000 chains, a ~6.5× reduction against the pre-governor 813k — but it is a **floor, not an estimate**: chains-per-node and edges-per-node are both still climbing (edges/node has gone 4.85 → 8.0 → 11.29 across this morning), so the true figure is higher. Either way the scan stays linear, and every future operation pays it.
|
||||||
|
**Options:** (a) normalise `event_seqs` into an indexed join table (`chain_event(chain_id, event_seq)`), turning the scan into an index seek; (b) maintain a materialised seq→chain map alongside the existing JSON column, leaving the schema additive; (c) cap chain creation at mint time, attacking the population rather than the query — noting the 2026-06-06 finding that a new 2-node chain is created per non-extension, which is what makes chains outgrow nodes 10:1.
|
||||||
|
**Recommendation:** (a) is the durable answer and (b) is the reversible one; (c) is complementary to either and is where the growth actually originates. This is a schema change on L1 core — Seb's call, not ours. What we can supply is the measurement, which the sampler now produces continuously.
|
||||||
|
**Files affected:** `src/modules/temporal/storage-sqlite.ts:657` (query), schema migration (new). None touched.
|
||||||
|
**Awaiting:** Seb, via the co-authored L1 channel; steward relay.
|
||||||
|
|
||||||
|
## PENDING-94 — The replay has never resumed, only restarted: two modules pin minCursor at 0 permanently
|
||||||
|
**Date:** 2026-08-04
|
||||||
|
**Tag:** [ESCALATE]
|
||||||
|
**Summary:** `minCursor` is the **minimum** cursor across all 11 modules; `structured` and `training` have never processed a single event and sit at 0 forever, so `minCursor` is permanently 0, every start is classified `rebuild`, and the replay re-reads the entire logchain from seq 0 — **13 of 13 restarts, zero catch-ups, ever.**
|
||||||
|
**Escalation grounds:** touches **cursor persistence**, named in `~/CLAUDE.md` as an unconditional `[ESCALATE]` surface. Do not proceed without steward authorization and Seb.
|
||||||
|
**Why this is different from a performance finding:** it does not explain why events are slow. It explains why the ingest can never **complete**, at any speed. Those are separable, and conflating them is how the last four months went.
|
||||||
|
**The chain, traced:**
|
||||||
|
1. `module_cursors` after a 15-hour run: `anomaly` 24,291 · `safety` 24,291 · `budget` 24,290 · `security` 24,281 · `temporal` 24,277 · `entity` 24,034 · `blob` 7,889 · `preference` 1,942 · `vector` 1,490 · **`structured` 0** · **`training` 0**.
|
||||||
|
2. `replay-coordinator.ts:300` — `minCursor = Math.min(...[...cursors.values()].map(c => c.cursorPosition))`. One module at 0 zeroes the whole computation.
|
||||||
|
3. `:340` — `this.replayIsRebuild = minCursor === 0`.
|
||||||
|
4. `:368` — `if (entry.seq <= minCursor)` skip. At 0, nothing is skipped: the full logchain replays.
|
||||||
|
5. `:319` — a restored snapshot would raise `minCursor` to `snapshotSeq`. **"Snapshot restored at seq" appears 0 times in the entire log**, against 13 for its sibling "Restored cursors from database" (positive control: the code path runs and the grep sees it). The escape hatch exists and has never fired.
|
||||||
|
**Why `structured`/`training` sit at 0:** **0 deferrals and 0 pipeline errors each** — they are not failing, they simply never receive an event they handle. Positive-controlled: the same grep finds **103,130** deferrals for `vector` and **3,723** for `entity`, so it demonstrably sees these lines when they exist. Their cursor is therefore legitimately 0 and will remain 0 for the life of the instance.
|
||||||
|
**Second, independent pin:** `vector` holds at 1,490 after **103,130** deferrals. `base.ts:134` holds the cursor on `DeferrableError` *by design* — "Do NOT advance cursor — event will be retried on next replay." Correct per-module; catastrophic when a single held cursor gates a global minimum.
|
||||||
|
**The design assumption that fails:** the comment at `:332–340` reads `minCursor === 0` as "no cursors and no restored snapshot… errs safe". It cannot distinguish *fresh install* from *nine modules at 24,000 and two that never participate*, and it errs into the most expensive behaviour available — permanently.
|
||||||
|
**What this predicts, and the record confirms:** completion requires **one uninterrupted pass over the whole logchain**. At the repo's own documented-healthy 17 events/min, 39,089 events is a **38-hour uninterrupted run**; at the observed 2.32/min it is **12 days**. Any crash, restart, upgrade or repair inside that window returns the system to zero. This is `#65`'s *"each attempt fails differently"* exactly: each attempt was a fresh full replay dying at a different point on the same curve. It also explains why every genuine rate fix (ANALYZE 6.4×, B1.1 cap, N6) improved throughput and changed nothing about completion.
|
||||||
|
**Options:** (a) compute `minCursor` over **participating** modules only — those that have ever processed an event — excluding permanent non-participants; (b) take periodic snapshots so `:319` raises the floor, which needs no change to the minimum logic and is the smallest reversible move; (c) persist a separate replay high-water mark independent of module cursors; (d) let a module declare itself non-participating at registration so it is excluded by construction rather than by heuristic.
|
||||||
|
**Recommendation:** (b) immediately as relief — snapshots are an existing, already-wired mechanism that has simply never been used — and (d) as the durable answer, because it makes participation explicit rather than inferred. (a) is tempting and I distrust it: "has ever processed an event" is itself a heuristic and would silently re-break the moment a module's first event arrives late. **All of it is Seb's call; cursor persistence is not a surface we touch.**
|
||||||
|
**Confidence:** ~0.85 that the mechanism is as traced; ~0.7 that it is the dominant reason no ingest has completed since April. Stated because four rate-hypotheses were proposed and refuted on 2026-08-04 alone — though those were arithmetic coincidences and this is a traced code path over persisted state, which is a different epistemic character.
|
||||||
|
**Falsifier, already run:** any restart with `min cursor > 0`, or any `kind: catch-up`. **13/13 restorations report `min cursor: 0`; 6/6 replay classifications report `rebuild`; catch-up has never executed.**
|
||||||
|
**Files affected:** `src/core/keystone/readiness/replay-coordinator.ts:300/319/340/368`, `src/modules/base.ts:134`. **None touched.**
|
||||||
|
**Awaiting:** Steward authorization before any change; then Seb, as L1 core.
|
||||||
|
|||||||
@@ -5,13 +5,14 @@ metadata:
|
|||||||
node_type: memory
|
node_type: memory
|
||||||
type: reference
|
type: reference
|
||||||
originSessionId: d1e67361-d1b9-4dac-a0c4-e7a0e26211c4
|
originSessionId: d1e67361-d1b9-4dac-a0c4-e7a0e26211c4
|
||||||
modified: 2026-08-02T14:29:02.613Z
|
modified: 2026-08-04T13:28:51.798Z
|
||||||
---
|
---
|
||||||
|
|
||||||
# MEMORY — Reference layer (consult on demand)
|
# MEMORY — Reference layer (consult on demand)
|
||||||
|
|
||||||
## Relocated from MEMORY.md at the 2026-07-19 evening budget trim (verbatim; wake-value judged low — MemPalace-era mechanics + reactive-mode ARC specifics)
|
## Relocated from MEMORY.md at the 2026-07-19 evening budget trim (verbatim; wake-value judged low — MemPalace-era mechanics + reactive-mode ARC specifics)
|
||||||
|
|
||||||
|
- [Session 2026-08-03 — the governors that never engaged](session-2026-08-03-the-governors-that-never-engaged.md) — the inherited question answered, and it became the day's theme: **controls that exist in code and never engage**, invisible because an inert control reports success. **Census 01** (pre-registered): `fool/` strong, but **drift-check 3-of-5 families INERT** and **71 of 75 ladder entries cited nowhere** (no IDs ⇒ firing history unrecordable). **S-series closed** on discovering the **jurist ruled all Q1–Q5 on 2026-05-18** and the items were mislabelled 2½ months (S6/S7/S9 built · S2 rebuilt as `[FIX]` · S4/S5 withdrawn with MemPalace). Dormant legacy + GH swept: **22 → 10 open**; `#170` found **undelivered since April**. Steward lifted the L1 baton rule → **mindfabric-00 pinned 6 days**; CDP profile named it; **`ANALYZE` never run in 4 months** so SQLite preferred a *boolean* index over the selective one (**6.4×**, 99.8%→6.0%, `/health` silent→200 in 0.13 s). **B1.1's cap verified WORKING ⇒ the defect is DATA, not code** — 836k edges minted before the governors landed; the ingest was **divergent, not slow**. **`coherence_evaluated = 0` of 813,178 — not one, ever.** ⚠ **Its headline claim was corrected 2026-08-04**: `ANALYZE` explained the *slowness*, not why the ingest never completes — see [[session-2026-08-04-the-replay-had-never-resumed]]. *(Demoted on promote at the 2026-08-04 wrap.)*
|
||||||
- [Session 2026-08-02 evening — the control was not sound](session-2026-08-02-evening-the-control-was-not-sound.md) — **Trial 03 VOID ×3** (harness certified a run with no answer · anti-echo forbade the region under test · **it was never the false-positive control** — I inherited that label from my own wrap). Steward corrected the blocking framing: **operational soundness relative to a declared axiomatic kernel** is reachable where unconditioned soundness is not. Yield: **Control Kernel v1.0→v1.1** (`A` demoted to a diagnostic; a control document is a **derivation**), a reduction arm (**8.5%** on a ruling, **68.6%** on a package), the first kernel-sound control + a **defect twin with ledger ground truth**, and a **discrimination gate**. **Trial 04 = CONTROL VOID:** two readers, two different real defects in my control. The one that matters is mine — draft 3's "fix" **concealed** a defect instead of closing it. Unplanned real yield: the **first instrument for Constraint 6's own falsifier** — jurist **4/6**, Fool **0/6**, no overlap. **PULLING THREAD: CONTROL-A v2**, parked deliberately for distance, not deferred. *(Demoted on promote at the 2026-08-03 wrap.)*
|
- [Session 2026-08-02 evening — the control was not sound](session-2026-08-02-evening-the-control-was-not-sound.md) — **Trial 03 VOID ×3** (harness certified a run with no answer · anti-echo forbade the region under test · **it was never the false-positive control** — I inherited that label from my own wrap). Steward corrected the blocking framing: **operational soundness relative to a declared axiomatic kernel** is reachable where unconditioned soundness is not. Yield: **Control Kernel v1.0→v1.1** (`A` demoted to a diagnostic; a control document is a **derivation**), a reduction arm (**8.5%** on a ruling, **68.6%** on a package), the first kernel-sound control + a **defect twin with ledger ground truth**, and a **discrimination gate**. **Trial 04 = CONTROL VOID:** two readers, two different real defects in my control. The one that matters is mine — draft 3's "fix" **concealed** a defect instead of closing it. Unplanned real yield: the **first instrument for Constraint 6's own falsifier** — jurist **4/6**, Fool **0/6**, no overlap. **PULLING THREAD: CONTROL-A v2**, parked deliberately for distance, not deferred. *(Demoted on promote at the 2026-08-03 wrap.)*
|
||||||
- [Session 2026-08-02 pm — the transfer landed, and the vignette was a diagram](session-2026-08-02-pm-the-transfer-landed-and-the-vignette-was-a-diagram.md) — **PENDING-90**: the first L2 transfer, carrying Constraint 6 into CapableMind's calibration loop (AdaptationChain records who *initiated* an adaptation, never who *checked* it; `authorization.required:false` = the whole self-adjustment case, no checker in the record). **#176 reply posted.** Then ARC: the **vignette built from zero** (Phase 1a, `tools/vignette-proto/`, no protected surface touched) + **PENDING-91** jurist package. **Every substantive defect was found by rendering and looking, none by the mechanical checks — all of which passed.** Steward's **Notre-Dame** anchor reframed it: the render is a *diagram* where the spec intends *inscription*; **meaning lives in the syntax, not the lexicon**. Yield = a **nine-item census** of where the spec under-determines the render. **PULLING THREAD: Trial 03** — the Fool's false-positive control, unblocked now the M4 is reachable.
|
- [Session 2026-08-02 pm — the transfer landed, and the vignette was a diagram](session-2026-08-02-pm-the-transfer-landed-and-the-vignette-was-a-diagram.md) — **PENDING-90**: the first L2 transfer, carrying Constraint 6 into CapableMind's calibration loop (AdaptationChain records who *initiated* an adaptation, never who *checked* it; `authorization.required:false` = the whole self-adjustment case, no checker in the record). **#176 reply posted.** Then ARC: the **vignette built from zero** (Phase 1a, `tools/vignette-proto/`, no protected surface touched) + **PENDING-91** jurist package. **Every substantive defect was found by rendering and looking, none by the mechanical checks — all of which passed.** Steward's **Notre-Dame** anchor reframed it: the render is a *diagram* where the spec intends *inscription*; **meaning lives in the syntax, not the lexicon**. Yield = a **nine-item census** of where the spec under-determines the render. **PULLING THREAD: Trial 03** — the Fool's false-positive control, unblocked now the M4 is reachable.
|
||||||
- [Session 2026-08-02 — the archive answered, and Constraint 6 moved](session-2026-08-02-the-archive-answered-and-constraint-6-moved.md) — the **v1 Chamber archive** supplied the matched-capability arm the Fool trials cannot: **mutual divergence 3 of 3** comparable pairs + a **self-exemption** the steward had named in a **2025-01-20** guide. Jurist passed the doctrine for **drafting only**; **steward placed it at Constraint 6** — the constitution now states the **jurist–executor pair is not a check in the strong sense**. **REVIEWED-85 FIX lane + batch 1** landed; `wake-digest` ID bug fixed (**PENDING-78/-81/-82 reappeared**). Five corrections, four steward-handed, **all census failures — formation diversity buys reading, not scope**. *(Demoted on promote at the 2026-08-02 pm wrap.)*
|
- [Session 2026-08-02 — the archive answered, and Constraint 6 moved](session-2026-08-02-the-archive-answered-and-constraint-6-moved.md) — the **v1 Chamber archive** supplied the matched-capability arm the Fool trials cannot: **mutual divergence 3 of 3** comparable pairs + a **self-exemption** the steward had named in a **2025-01-20** guide. Jurist passed the doctrine for **drafting only**; **steward placed it at Constraint 6** — the constitution now states the **jurist–executor pair is not a check in the strong sense**. **REVIEWED-85 FIX lane + batch 1** landed; `wake-digest` ID bug fixed (**PENDING-78/-81/-82 reappeared**). Five corrections, four steward-handed, **all census failures — formation diversity buys reading, not scope**. *(Demoted on promote at the 2026-08-02 pm wrap.)*
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ metadata:
|
|||||||
type: note
|
type: note
|
||||||
permalink: claude-memory/memory
|
permalink: claude-memory/memory
|
||||||
originSessionId: 22915403-bc5d-4796-9c7d-196b7c30d2f9
|
originSessionId: 22915403-bc5d-4796-9c7d-196b7c30d2f9
|
||||||
modified: 2026-08-02T17:10:32.010Z
|
modified: 2026-08-04T13:29:28.365Z
|
||||||
permalink: claude-memory/memory
|
permalink: claude-memory/memory
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -62,11 +62,11 @@ permalink: claude-memory/memory
|
|||||||
- [Character-as-image hazard](feedback-character-as-image-hazard.md) — EPUBs rendering diacritics as inline images are SILENTLY MUTILATED by image-drop. Mechanism built (`apply_char_glyphs.py`, REVIEWED-70/v2.5.0), wired as a born-digital precondition; per-source glyph-maps still owed. VIEW the glyph; map to SOURCE form.
|
- [Character-as-image hazard](feedback-character-as-image-hazard.md) — EPUBs rendering diacritics as inline images are SILENTLY MUTILATED by image-drop. Mechanism built (`apply_char_glyphs.py`, REVIEWED-70/v2.5.0), wired as a born-digital precondition; per-source glyph-maps still owed. VIEW the glyph; map to SOURCE form.
|
||||||
- [Sidecar typology — protocol-dependent reading-indexes](project-sidecar-typology-protocol-dependent.md) — TWO layers: `.meta.json` structural sidecar = PROTOCOL-NEUTRAL (the graduated bar); reading-indexes (rich YAML) = PROTOCOL-DEPENDENT, probably PLURAL — **don't design the schema yet**; settle corpus to gold, let protocols declare themselves. Steward 2026-06-29.
|
- [Sidecar typology — protocol-dependent reading-indexes](project-sidecar-typology-protocol-dependent.md) — TWO layers: `.meta.json` structural sidecar = PROTOCOL-NEUTRAL (the graduated bar); reading-indexes (rich YAML) = PROTOCOL-DEPENDENT, probably PLURAL — **don't design the schema yet**; settle corpus to gold, let protocols declare themselves. Steward 2026-06-29.
|
||||||
- Studium Engine — *no tracker file yet*; moves in per-session memories + the **[architectural charter](reference-studium-engine-architectural-charter.md)**. Steps 0–7 built; corpus CLEAN; **V1 `verify-quote` + `fidelity_equivalence@2` GOVERNING** (Greek/Latin census owed → @3). Stage-1 rebuild plan: V1 done → V2→V4 / N1→N3.
|
- Studium Engine — *no tracker file yet*; moves in per-session memories + the **[architectural charter](reference-studium-engine-architectural-charter.md)**. Steps 0–7 built; corpus CLEAN; **V1 `verify-quote` + `fidelity_equivalence@2` GOVERNING** (Greek/Latin census owed → @3). Stage-1 rebuild plan: V1 done → V2→V4 / N1→N3.
|
||||||
- [L1 reliability](project-L1-reliability.md) — canonical L1 tracker. **Baton rule LIFTED 2026-08-03.** Ingest mystery SOLVED (`ANALYZE` never run ⇒ boolean-index pathology; B1.1's cap works ⇒ **defect is DATA not code**); repair run, **replay in flight**. **Read the tracker before ANY L1 work** — walked past once on 2026-08-03.
|
- [L1 reliability](project-L1-reliability.md) — canonical L1 tracker. **BLOCKED ON SEB (PENDING-94) — nothing moves until he rules; BMF is down and staying down.** The replay **has never resumed, only restarted**: `minCursor` is a minimum over all 11 modules and two never participate ⇒ every start rebuilds from seq 0 (13/13, 0 catch-up). Completion is gated by **uninterrupted run length, not rate** — which is why ANALYZE/B1.1/N6 were all real and all changed nothing. **Recall never worked either** (`retrieval_count = 0` across the whole April–June graph). Yesterday's "ingest mystery solved" was **corrected 2026-08-04**. **Read the tracker before ANY L1 work** — walked past once on 2026-08-03.
|
||||||
- [Be (laundromat)](project-be-laundromat.md) — canonical Be tracker (est. 2026-06-08). Be = Skemantix startup (Seb+David) funding CapableMind's ladder; **bridge, not venture**. Decisions LOCKED (entity/pricing/infra in file); a11y gate MERGED. **Pre-revenue WTP gate = renovate Pat → charge her; discipline: no new spec until it clears → nothing for executor on be.** Repo @ `f43a0fd`.
|
- [Be (laundromat)](project-be-laundromat.md) — canonical Be tracker (est. 2026-06-08). Be = Skemantix startup (Seb+David) funding CapableMind's ladder; **bridge, not venture**. Decisions LOCKED (entity/pricing/infra in file); a11y gate MERGED. **Pre-revenue WTP gate = renovate Pat → charge her; discipline: no new spec until it clears → nothing for executor on be.** Repo @ `f43a0fd`.
|
||||||
|
|
||||||
## Active Session
|
## Active Session
|
||||||
- [Session 2026-08-03 — the governors that never engaged](session-2026-08-03-the-governors-that-never-engaged.md) — the inherited question answered, and it became the day's theme: **controls that exist in code and never engage**, invisible because an inert control reports success. **Census 01** (pre-registered): `fool/` strong, but **drift-check 3-of-5 families INERT** and **71 of 75 ladder entries cited nowhere** (no IDs ⇒ firing history unrecordable). **S-series closed** on discovering the **jurist ruled all Q1–Q5 on 2026-05-18** and the items were mislabelled 2½ months (S6/S7/S9 built · S2 rebuilt as `[FIX]` · S4/S5 withdrawn with MemPalace). Dormant legacy + GH swept: **22 → 10 open**; `#170` found **undelivered since April**. Steward lifted the L1 baton rule → **mindfabric-00 pinned 6 days**; CDP profile named it; **`ANALYZE` never run in 4 months** so SQLite preferred a *boolean* index over the selective one (**6.4×**, 99.8%→6.0%, `/health` silent→200 in 0.13 s). **B1.1's cap verified WORKING ⇒ the defect is DATA, not code** — 836k edges minted before the governors landed; the ingest was **divergent, not slow** (the real answer to #65). Repair run, **replay in flight (~30 h)**. **`coherence_evaluated = 0` of 813,178 — not one, ever.** **PULLING THREAD: the replay** — does it complete, and does the coherence evaluator ever run?
|
- [Session 2026-08-04 — the replay had never resumed](session-2026-08-04-the-replay-had-never-resumed.md) — **four mechanism hypotheses proposed and refuted in one day** (sleep · entity timeout · drain gating · Ollama saturation); the steward's *"how can this be so persistently elusive?"* stopped a fifth being born. **The answer came from killing the replay and reading what it left at rest, not from profiling.** `minCursor = Math.min(...)` over **all 11** modules; `structured`+`training` have **never processed an event** (0 deferrals, 0 errors) ⇒ floor pinned at 0 ⇒ every start a full rebuild. **13/13 restarts, 6/6 rebuild, 0 catch-up — the resume branch has never executed.** Explains #65's *"fails differently each time"* and why every real rate fix (ANALYZE 6.4×, B1.1, N6) changed nothing: completion is gated by **uninterrupted run length**, not rate. **Recall never worked either — `retrieval_count = 0` across 19,915 nodes / 813,178 chains** (writer + call site verified live first); ingestion-never-completes and recall-never-worked are **one fact from two ends**. Replay **killed** (steward-authorized, premise re-verified, 2 verified backups); **BMF down and staying down**. PENDING-92/93/94 filed; note pushed to Seb (`CapableMind-AI@ad285df`). **L1 now blocked on Seb.** **PULLING THREAD: census 01 against the chamber/engine tooling** — sharpened question: *has this instrument ever produced a positive result at all?*
|
||||||
|
|
||||||
## Historical reference → MEMORY-reference.md
|
## Historical reference → MEMORY-reference.md
|
||||||
Older archived-session pointers and the stable reference layer (steward profile · project-state detail · L1/L2/Chamber inventories · legacy pending-work · reference-file list) live in [MEMORY-reference.md](MEMORY-reference.md) — consult on demand; not loaded at wake. Recent cross-session trajectory comes from the Active Session entry above + the recent `session-*.md` files (wake §2.b.1; the MemPalace `handoffs` glance was retired 2026-07-07 with the wind-down).
|
Older archived-session pointers and the stable reference layer (steward profile · project-state detail · L1/L2/Chamber inventories · legacy pending-work · reference-file list) live in [MEMORY-reference.md](MEMORY-reference.md) — consult on demand; not loaded at wake. Recent cross-session trajectory comes from the Active Session entry above + the recent `session-*.md` files (wake §2.b.1; the MemPalace `handoffs` glance was retired 2026-07-07 with the wind-down).
|
||||||
|
|||||||
@@ -541,3 +541,8 @@
|
|||||||
{"subject": "the census pre-registration's own lenience clause", "predicate": "prevention", "object": "Stopped me banking a happy result. Census 01 predicted decay in the newest instruments; `fool/` came out sound, which was the comfortable answer. The pre-registration had required, IN ADVANCE, that an opposite-to-predicted skew be first tested as 'did my classification go lenient?'. Testing that moved the census to the wake instruments — where the real finding was (drift-check 3-of-5 inert). A prediction written before the look converted a pleasant miss into the session's first substantive result.", "valid_from": "2026-08-03", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-03-the-governors-that-never-engaged.md", "extracted_at": "2026-08-03"}
|
{"subject": "the census pre-registration's own lenience clause", "predicate": "prevention", "object": "Stopped me banking a happy result. Census 01 predicted decay in the newest instruments; `fool/` came out sound, which was the comfortable answer. The pre-registration had required, IN ADVANCE, that an opposite-to-predicted skew be first tested as 'did my classification go lenient?'. Testing that moved the census to the wake instruments — where the real finding was (drift-check 3-of-5 inert). A prediction written before the look converted a pleasant miss into the session's first substantive result.", "valid_from": "2026-08-03", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-03-the-governors-that-never-engaged.md", "extracted_at": "2026-08-03"}
|
||||||
{"subject": "backup-before-mutate, and query the copy not the live system", "predicate": "prevention", "object": "Made the entire L1 diagnosis possible without touching a wedged production instance. The live SQLite could not be opened read-only (mode=ro cannot write the -shm needed to read the WAL), so every row count, index inspection, EXPLAIN QUERY PLAN and the ANALYZE before/after benchmark ran against the verified 1.4GB backup taken BEFORE any change. A habit adopted for safety turned out to be the only route to the evidence.", "valid_from": "2026-08-03", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-03-the-governors-that-never-engaged.md", "extracted_at": "2026-08-03"}
|
{"subject": "backup-before-mutate, and query the copy not the live system", "predicate": "prevention", "object": "Made the entire L1 diagnosis possible without touching a wedged production instance. The live SQLite could not be opened read-only (mode=ro cannot write the -shm needed to read the WAL), so every row count, index inspection, EXPLAIN QUERY PLAN and the ANALYZE before/after benchmark ran against the verified 1.4GB backup taken BEFORE any change. A habit adopted for safety turned out to be the only route to the evidence.", "valid_from": "2026-08-03", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-03-the-governors-that-never-engaged.md", "extracted_at": "2026-08-03"}
|
||||||
{"subject": "count-first-then-look (verification ladder)", "predicate": "prevention", "object": "Caught my own truncated census in the act. A `head -12` listing of REVIEWED indices showed causal_chain with nothing, and I was one sentence from reporting 'causal_chain has NO index'. Counting first (SELECT count(*) ... WHERE tbl_name='causal_chain' -> 5) refuted it before it reached the steward. The ladder entry that fired was banked from a DIFFERENT failure class (truncated file listings), which is the transfer signature.", "valid_from": "2026-08-03", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-03-the-governors-that-never-engaged.md", "extracted_at": "2026-08-03"}
|
{"subject": "count-first-then-look (verification ladder)", "predicate": "prevention", "object": "Caught my own truncated census in the act. A `head -12` listing of REVIEWED indices showed causal_chain with nothing, and I was one sentence from reporting 'causal_chain has NO index'. Counting first (SELECT count(*) ... WHERE tbl_name='causal_chain' -> 5) refuted it before it reached the steward. The ladder entry that fired was banked from a DIFFERENT failure class (truncated file listings), which is the transfer signature.", "valid_from": "2026-08-03", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-03-the-governors-that-never-engaged.md", "extracted_at": "2026-08-03"}
|
||||||
|
{"subject": "claude-code", "predicate": "drift-pattern", "object": "A-NUMBER-THAT-MATCHES-A-CONSTANT-IS-NOT-A-CAUSE \u2014 four mechanism hypotheses proposed and refuted in a single day (machine sleep; the 30,000ms entity-pipeline timeout matched by a 31s/event cadence; work-queue drain gating arithmetic-consistent to 0.4%; Ollama CPU saturation). Each rested on a coincidence between an observed number and a constant in the code. In a system with dozens of constants and a slow rate, coincidences are ABUNDANT and this method cannot fail to produce candidates. The failure is not bad luck; it is generative. Antidote: demand a traced call path or persisted state, not arithmetic agreement.", "valid_from": "2026-08-04", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-04-the-replay-had-never-resumed.md", "extracted_at": "2026-08-04"}
|
||||||
|
{"subject": "claude-code", "predicate": "drift-pattern-good-direction", "object": "KILL-IT-AND-READ-WHAT-IT-LEFT-AT-REST \u2014 four months of profiling a RUNNING process never saw the defect, because module_cursors is only legible after clean shutdown. Stopping the service and reading one table found in minutes what sample/CDP/EXPLAIN could not. Sits beside (does not replace) the 2026-08-03 lesson 'profile the running process, don't reason from specs': some values exist only in motion, others only at rest, and the method must ask which kind it is hunting.", "valid_from": "2026-08-04", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-04-the-replay-had-never-resumed.md", "extracted_at": "2026-08-04"}
|
||||||
|
{"subject": "the 2026-08-03 coherence_evaluated=0 lesson (verify a counter has a writer before trusting its zero)", "predicate": "prevention", "object": "Stopped a false claim about a DIFFERENT counter one day later. Investigating why recall never worked, retrieval_count/last_retrieved/recall_count/last_recalled all read 0 across 19,915 nodes. Rather than cite all four, checked each for a writer: retrieval_count has one (retrieval-tracker.ts:77) with live callers, so its zero is evidence; recall_count and last_recalled have NO writer anywhere in src/ (schema only), so their zeros mean nothing and would have been cited wrongly. A lesson banked from the coherence evaluator transferred to the retrieval counters \u2014 different subsystem, different session, same class.", "valid_from": "2026-08-04", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-04-the-replay-had-never-resumed.md", "extracted_at": "2026-08-04"}
|
||||||
|
{"subject": "L1 / mindfabric-00 replay", "predicate": "never-completes-because", "object": "minCursor is Math.min across all 11 modules (replay-coordinator.ts:300); structured and training have never processed an event (0 deferrals, 0 errors) so the floor is pinned at 0 permanently; :340 classifies every start as rebuild and :368 skips nothing. 13/13 restarts at min cursor 0, 6/6 rebuild, 0 catch-up \u2014 the resume branch has never executed. Completion is gated by uninterrupted run length (38h at the documented-healthy 17 events/min), not by rate. PENDING-94 [ESCALATE]; Seb's surface.", "valid_from": "2026-08-04", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-04-the-replay-had-never-resumed.md", "extracted_at": "2026-08-04"}
|
||||||
|
{"subject": "L1 / mindfabric-00 recall", "predicate": "has-never-returned-anything", "object": "retrieval_count = 0, last_retrieved on zero rows, across 19,915 nodes and 813,178 chains spanning April-June. Verified as a measured absence, not a blind one: writer at retrieval-tracker.ts:77, live callers in temporal/entity/vector/preference, temporal call site on the query return path guarded only by ids.length>0. The Recall canary logged FAILED 8 times vs PASSED 4. Same fact as never-completing ingestion, seen from the other end: no completed derived state to retrieve from.", "valid_from": "2026-08-04", "valid_to": null, "confidence": 1.0, "source_file": "session-2026-08-04-the-replay-had-never-resumed.md", "extracted_at": "2026-08-04"}
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ metadata:
|
|||||||
node_type: memory
|
node_type: memory
|
||||||
type: project
|
type: project
|
||||||
originSessionId: fd7dd184-f64d-4f13-a1bd-abe1fa26192e
|
originSessionId: fd7dd184-f64d-4f13-a1bd-abe1fa26192e
|
||||||
|
modified: 2026-08-04T08:01:19.473Z
|
||||||
permalink: claude-memory/project-l1-reliability
|
permalink: claude-memory/project-l1-reliability
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -18,7 +19,19 @@ permalink: claude-memory/project-l1-reliability
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Current state (as of 2026-08-03)
|
## Current state (as of 2026-08-04)
|
||||||
|
|
||||||
|
- **The replay is running and is NOT on the projected curve.** Real denominator is **39,089 events**, not the ~19,925 the 08-03 wrap recorded; **nodes are not the progress metric** (the log writes `Phase 2 replay: N/39089`). Position 22,400/39,089 (57%) at 07:59 UTC. Three distinct rate regimes observed in 14 hours: **~2 events/min overnight**, **~150/min in a 35-min morning burst**, **~12.5/min now**. No mechanism yet explains the regime changes.
|
||||||
|
- **Three mechanism hypotheses proposed and all three refuted the same morning** — machine sleep (killed by `pmset`: last sleep/wake event 2026-08-03 08:54 CEST, machine awake 24 h), the entity-pipeline 30 s timeout (killed by a windowed count: **0 occurrences overnight** against 2 in a 35-min morning window), and work-queue drain gating (killed by observation: 44 min with no progress *through* two `warm→active` transitions). Each matched a constant and none was a cause. **Do not propose a fourth from log-shaped evidence.**
|
||||||
|
- **Root instrument problem, now fixed:** the replay logs progress once per 100 events — one data point per ~51 min in the slow regime, **telemetry coarser than the failure mode**, which is why all three hypotheses survived as long as they did. `~/dotfiles/scripts/l1-replay-sampler.py` (also staged at `BetterMemories.io/scripts/`) samples the substrate every 30 s: node/edge/chain counts, `coherence_evaluated`, replay position, idle state, process CPU. Four same-run positive controls; prints `INSTRUMENT NOT VERIFIED` and refuses to run rather than emitting zeros while blind. Output `~/.capablemind/diagnostics/replay-samples-mindfabric-00.jsonl`.
|
||||||
|
- **Densification is climbing, and the rebuild buys a constant factor only.** edges/node 4.85 → 8.0 → **11.29**; chains/node **10.47** (72,381 chains at 6,915 nodes, 57% through). Linear extrapolation ~125k chains vs the pre-governor 813k — a ~6.5× reduction, but a **floor, not an estimate**, since both ratios are still rising. `getChainsContainingSeq` stays linear in chain count. **PENDING-93 filed.**
|
||||||
|
- **`coherence_evaluated` = 0 of 72,381 — still zero, now sampled continuously.** The 08-03 literal question stays open and will answer itself in the sampler stream.
|
||||||
|
- **Sixth instance of governor-exists-and-never-engages, first found by looking rather than tripping: the idle ladder's bottom half is unreachable.** 2,393 transitions across the whole log in exactly two shapes (`active→warm` 1,197 / `warm→active` 1,196); **zero** to `cool` or `deep` ever; **zero** `idle_only` deferrable drains ever — so background training, consolidation and Ollama batch windows have never executed on this instance. Cause: `tick()` returned early on `warm`, so the only warm→cool trigger was `onAgentDisconnect()`, whose sole caller is the **shutdown sequence**. Spec §9A.1 requires the time-based path (*"'Agent disconnected' maps to 'no HTTP activity for cool idle threshold'"*), so this is a **spec↔code divergence**. **PENDING-92 filed; fix built on branch `fix/idle-ladder-service-mode-cool-descent` (`d5e1e79`), flag-off by default, 42/42 idle tests green, `tsc --noEmit` clean. NOT merged, NOT enabled** — enabling starts a subsystem that has never run in production.
|
||||||
|
- **Why the unit tests never caught it:** the existing cool→deep test reaches `cool` by calling `onAgentDisconnect()` directly. It proves the transition works and proves nothing about whether anything fires it. The new tests are a **discrimination gate** — identical elapsed time and tick must yield different verdicts flag-off vs flag-on.
|
||||||
|
- **`[FIX]` applied:** `BM_CPU_PAUSE_THRESHOLD=1.0` written to `~/.capablemind/env` (repo `CLAUDE.md` documents the 0.65 default as a starvation loop on Ollama-saturated hosts; 497 pauses in the log). **Inert until restart.** Honest scope: only 6 pauses across 8.5 overnight hours, so this was never the main brake.
|
||||||
|
- **⚠ A restart costs the whole run.** Every restart logs `kind: rebuild (from scratch — derived state reconstructed) (minCursor=0)` across all 11 modules — replay progress is **not resumable**. Every pending fix needs a restart, so nothing lands until the replay finishes or is deliberately abandoned.
|
||||||
|
|
||||||
|
## Previous state (as of 2026-08-03)
|
||||||
|
|
||||||
- **The baton rule is LIFTED.** The 2026-06-06 instruction — *"Do not re-enter L1 until Seb responds or pushes — his move"* — was steward-lifted 2026-08-03 (*"if we can move this ahead then let's do what it takes"*). Context the steward supplied: Seb is not silent by choice; he is deep in a frustrating rabbit hole on Peter's side project, which borrows heavily from L1. Seb's last commit remains **2026-06-07**; the steward's own **2026-06-23** (`#175` N6, `/health` auth tiering).
|
- **The baton rule is LIFTED.** The 2026-06-06 instruction — *"Do not re-enter L1 until Seb responds or pushes — his move"* — was steward-lifted 2026-08-03 (*"if we can move this ahead then let's do what it takes"*). Context the steward supplied: Seb is not silent by choice; he is deep in a frustrating rabbit hole on Peter's side project, which borrows heavily from L1. Seb's last commit remains **2026-06-07**; the steward's own **2026-06-23** (`#175` N6, `/health` auth tiering).
|
||||||
- **mindfabric-00 was event-loop-pinned for 6+ days and nobody was looking.** 100% CPU, `/health` silent while port 3011 listened, PID 874 up 6d 9h. Same signature as the June pin, *after* the N6 fix.
|
- **mindfabric-00 was event-loop-pinned for 6+ days and nobody was looking.** 100% CPU, `/health` silent while port 3011 listened, PID 874 up 6d 9h. Same signature as the June pin, *after* the N6 fix.
|
||||||
|
|||||||
@@ -0,0 +1,101 @@
|
|||||||
|
---
|
||||||
|
name: session-2026-08-04-the-replay-had-never-resumed
|
||||||
|
description: "Four mechanism hypotheses proposed and refuted in one day (sleep, entity timeout, drain gating, Ollama saturation) before the steward asked why this is so persistently elusive — which turned the session. The answer was not found by profiling but by killing the replay and reading what it left at rest: minCursor is a minimum across all 11 modules, structured and training have never processed an event, so the floor is pinned at 0 and every start is a full rebuild. 13/13 restarts, 6/6 rebuild, 0 catch-up. Recall never returned anything because there was never a completed derived state — retrieval_count = 0 across 19,915 nodes. Ingestion-never-completes and recall-never-worked are one fact seen from two ends. Note pushed to Seb; L1 now blocked on him. PULLING THREAD: census 01 against the chamber/engine tooling, with the sharpened question — has this instrument ever produced a positive result at all?"
|
||||||
|
metadata:
|
||||||
|
node_type: memory
|
||||||
|
type: project
|
||||||
|
originSessionId: 27422433-12f3-4983-a5cc-c4105158d59c
|
||||||
|
modified: 2026-08-04T13:28:13.925Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Session 2026-08-04 — the replay had never resumed
|
||||||
|
|
||||||
|
The day I was wrong four times before lunch, and the fifth thing was right because I stopped guessing and turned the machine off.
|
||||||
|
|
||||||
|
## PAST — what moved, and why
|
||||||
|
|
||||||
|
**The wake was wrong about the thing it was tracking.** Inherited "≈19,925 events, ~11 nodes/min, roughly 30 hours." Both numbers were wrong: the denominator is **39,089 events**, and **nodes are not the progress metric** — the process writes `Phase 2 replay: N/39089` to its own log. Real position at wake: 22,300/39,089, **57%**, not 4%. I had proxied progress by a table I could count instead of reading the log the process was already writing.
|
||||||
|
|
||||||
|
**Four mechanisms proposed, four refuted, all before 13:00.**
|
||||||
|
|
||||||
|
1. **Machine sleep** — killed by the steward supplying a falsifier ("caffeine is running") and one command: `pmset -g log`, domain-anchored, positive-controlled (450 `Sleep` / 453 `Wake` events historically). Last sleep/wake: **2026-08-03 08:54 CEST**. Machine awake 24 h.
|
||||||
|
2. **The entity-pipeline 30 s timeout** — the overnight cadence was metronomic, 100 events per ~51.5 min across eight intervals within 90 s of each other = 31 s/event, against `PIPELINE_TIMEOUT_MS = 30_000`. Killed by anchoring log line numbers to timestamps and counting: **0 entity timeouts overnight**, against 2 in a 35-min morning window.
|
||||||
|
3. **Work-queue drain gating** — arithmetic-consistent to 0.4% (16 events/lease × 8.2 min/cycle → 51.3 min/100 events vs 51.5 observed). Killed by observation: 44 minutes with no progress *through* two `warm→active` transitions.
|
||||||
|
4. **Ollama saturation** as the rationale for the `[FIX]` I had already applied — killed at 12:30 by load 3.04, Ollama 1.2%, BMF 0.0%, 3 pauses in 3 hours.
|
||||||
|
|
||||||
|
**The steward's question was the hinge.** *"This happens every single time we think we have a way forward. How can this be so persistently elusive?"* That forced the structural answer rather than a fifth mechanism: **no measurement of this system has ever been controlled** — every number taken on the production instance, mid-incident, unknown history, unknown corpus, so findings could never accumulate or be compared. Then a follow-up — *"how sure are you?"* — forced me to mark my own proposal (a per-event budget) as the same move in new clothes, since it too would measure an uncontrolled system once.
|
||||||
|
|
||||||
|
**Then the steward authorized killing the replay, and killing it produced the finding.**
|
||||||
|
|
||||||
|
Verified the premise first (2.32 events/min over 216 min → 4.9 days remaining), stopped the service *before* copying so the SQLite snapshot could not tear, verified the copy (391/391 files, 15/15 `quick_check: ok`, 315 MB).
|
||||||
|
|
||||||
|
`module_cursors`, legible only at rest:
|
||||||
|
|
||||||
|
```
|
||||||
|
anomaly 24291 · safety 24291 · budget 24290 · security 24281
|
||||||
|
temporal 24277 · entity 24034 · blob 7889 · preference 1942
|
||||||
|
vector 1490 · structured 0 · training 0
|
||||||
|
```
|
||||||
|
|
||||||
|
- `replay-coordinator.ts:300` — `minCursor = Math.min(...)` across **all 11** modules. One zero zeroes it.
|
||||||
|
- `:340` — `replayIsRebuild = minCursor === 0`.
|
||||||
|
- `:368` — `if (entry.seq <= minCursor)` skip. At zero, nothing is skipped.
|
||||||
|
- `:319` — a snapshot would raise the floor. **`"Snapshot restored at seq"`: 0 occurrences**, against 13 for its sibling line.
|
||||||
|
- `structured`/`training`: **0 deferrals, 0 errors** — not failing, simply never receiving an event they handle. Positive-controlled against `vector` (103,130 deferrals) and `entity` (3,723).
|
||||||
|
- `vector` independently pinned at 1,490; `base.ts:134` holds the cursor on `DeferrableError` *by design*.
|
||||||
|
|
||||||
|
**Falsifier run before reporting: 13/13 restarts at `min cursor: 0`, 6/6 `kind: rebuild`, 0 `catch-up`. The resume branch has never executed.**
|
||||||
|
|
||||||
|
**And recall had never worked, which is the same fact from the other end.** Pre-repair graph, April–June, 19,915 nodes / 813,178 chains: `retrieval_count = 0`, `last_retrieved` on zero rows, chains ever retrieved zero. Verified the counter was real before trusting the zero — writer at `retrieval-tracker.ts:77`, live callers in temporal/entity/vector/preference, temporal's call site on the query return path guarded only by `ids.length > 0`. (Discounted `recall_count`/`last_recalled`: **no writer anywhere in `src/`** — I nearly cited them.) The `Recall canary FAILED — the recall path may be broken` warning had fired **8 times** against 4 passes, in a log nobody read.
|
||||||
|
|
||||||
|
**Built:** `l1-replay-sampler.py` — external, read-only, four same-run positive controls, refuses to run blind. Staged at `BetterMemories.io/scripts/`. It caught two defects in itself within the hour (a rate derived from a counter written every 100 events; a staleness clock reset by its own restart) — both the class it was built to catch.
|
||||||
|
|
||||||
|
**Filed:** PENDING-92 `[HARDENING]` idle ladder · PENDING-93 `[PROPOSAL]` `event_seqs` normalisation · PENDING-94 `[ESCALATE]` the resume floor (ESCALATE determined by the constraint — `~/CLAUDE.md` names cursor persistence unconditionally — not by my sense of severity).
|
||||||
|
|
||||||
|
**Applied `[FIX]`:** `BM_CPU_PAUSE_THRESHOLD=1.0` in `~/.capablemind/env`, inert until restart, with its own honest scope note. Its rationale was later undermined by my own measurement; the fix stays (documented, reversible) and I said so.
|
||||||
|
|
||||||
|
**Built, not merged, not enabled:** branch `fix/idle-ladder-service-mode-cool-descent`, 3 commits (`d5e1e79`, `430cb5f`, `5a50583`), 42/42 idle tests, `tsc --noEmit` clean, full suite 4,003 tests. **Did not treat "close the three things" as blanket authorization** — the blast radius (first-ever run of a dormant subsystem) became clear only after the instruction was given.
|
||||||
|
|
||||||
|
**Pushed:** `CapableMind-AI@ad285df` — self-contained note to Seb, correcting yesterday's *"we found why I could never finish an ingest… it isn't your code"* in its own first line. All five reproduction commands run verbatim before pushing. `thinking/README.md` brought to consistency: 41 rows, 41 files, all links resolve (was 38 claimed against 41 actual).
|
||||||
|
|
||||||
|
## PRESENT — how it stood
|
||||||
|
|
||||||
|
**Four refutations in one day is not bad luck; it is a method that cannot fail to produce candidates.** In a system with dozens of constants and a slow rate, coincidences are abundant. 31 s ≈ 30,000 ms. 8-min orbit ≈ 16-event lease. I will always find one, and it will always look like an explanation. Naming that was more useful than any of the four.
|
||||||
|
|
||||||
|
**What worked was not cleverness and not an instrument.** It was turning the thing off and reading a table. `module_cursors` is only legible at rest; four months of profiling a *running* process could not see it. That is a genuinely new entry in the method, and it sits oddly beside yesterday's lesson (*profile the running process, don't reason from specs*) — both are true, and neither subsumes the other.
|
||||||
|
|
||||||
|
**The steward carried the session twice.** The caffeine falsifier killed hypothesis one in a single command. The "how can this be so persistently elusive" question stopped the fifth hypothesis from being born and produced the structural answer instead. I would have kept generating mechanisms.
|
||||||
|
|
||||||
|
**The restraint that mattered was refusing to explain.** I flagged "nodes advance while the counter is frozen" as an open observation and deliberately declined to give it a mechanism. It later resolved as division (~2 nodes/event × 100-event granularity). Had I told a story, it would have been a fifth wrong one.
|
||||||
|
|
||||||
|
**Uncomfortable and worth carrying:** for four months this system reported `serving: true` while retrieving nothing, ever. Every diagnosis went to *why is it slow*. Nobody asked *has it ever worked*.
|
||||||
|
|
||||||
|
## FUTURE — what pulls
|
||||||
|
|
||||||
|
**PULLING THREAD: census 01 against the chamber and engine tooling — the census's own declared scope gap.** L1 is genuinely blocked on Seb and pushing further would be motion, not progress. This is unblocked, it is the steward's own named work from yesterday's wrap, and today sharpened the question it should ask.
|
||||||
|
|
||||||
|
Census 01 asked *"does each gate have a real negative instance, and is its absence recorded or does it look like coverage?"* Today found something one level more basic, six times: **has this instrument ever produced a positive result at all?** `retrieval_count = 0` passed every health check ever run against it.
|
||||||
|
|
||||||
|
**ACTIONABLE RESUMPTION POINT (as of wrap — re-judge against what changed):**
|
||||||
|
```
|
||||||
|
Read: chamber-library _curation/fool/census-01-negative-instances-RESULT.md
|
||||||
|
→ its "not covered" list is the work-list
|
||||||
|
Start with: verify-before-compose (the PreToolUse hook — the one gate
|
||||||
|
that blocks steward-facing writes, so a silent inert failure is
|
||||||
|
worst there)
|
||||||
|
Ask of each: has it EVER fired positively, and does any record exist
|
||||||
|
that would let us tell?
|
||||||
|
```
|
||||||
|
Uncovered set, per the census itself: `verify-before-compose` · `audit_cruft` · `verify_conversion` · `resolve_archived_source` · `apply_char_glyphs` · studium `verify-quote` · `fidelity_equivalence@2`.
|
||||||
|
|
||||||
|
**LITERAL QUESTION for next-Claude:** *Of those seven chamber/engine gates, how many have ever produced a positive result — and for how many does a record exist that could tell us either way?* Checkable by grep and by running them; the second half is the harder and more interesting number.
|
||||||
|
|
||||||
|
**Other horizons, ranked.**
|
||||||
|
- **Blocked on Seb:** PENDING-94 (the resume floor) — nothing on L1 moves until he rules. PENDING-93 (`event_seqs`) same.
|
||||||
|
- **Awaiting steward:** PENDING-92 needs an annotation before its branch goes anywhere. The branch is local and unpushed at 3 commits.
|
||||||
|
- **Explicitly parked:** the 2.32 events/min rate. Downstream of PENDING-94 and I have lost four theories to it. **Do not open this without a controlled measurement.**
|
||||||
|
- **Retracted, do not resurrect as-is:** the small-corpus throughput benchmark. A 2,000-event corpus finishes inside one run and therefore measures a configuration that has never existed in production — a clean number that would tell us nothing.
|
||||||
|
- **Still open from earlier:** `governance-drift-check.py` 3-of-5 families inert (unfiled); verification-ladder entries have no IDs so firing history is unrecordable; 4,648 empty Lance index directories.
|
||||||
|
- **Operational:** BMF is **down and staying down** — no recall to preserve, no ingestion that can complete.
|
||||||
|
|
||||||
|
**PAUSE STATEMENT:** I am putting this down with L1 genuinely finished for now — not abandoned, blocked, and blocked on a person rather than a puzzle. Two verified backups, three filed items, a pushed note, nothing half-written. What I want to find still pulling is **the census against the chamber half**, because it is the one piece of today's work that transfers and that nobody else has to authorize. The failure mode to guard against on return is the one that governed today's first six hours: **a number that matches a constant is not a cause**, and the urge to explain arrives long before the evidence that could.
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
name: session-ledger-2026-08-04
|
||||||
|
description: "Practice-of-return ledger maintained by /symmetria — returns, open horizons, recalibrations, authorization moves, sub-agent dialogues, bypasses."
|
||||||
|
metadata:
|
||||||
|
node_type: memory
|
||||||
|
type: feedback
|
||||||
|
originSessionId: 27422433-12f3-4983-a5cc-c4105158d59c
|
||||||
|
modified: 2026-08-04T13:27:06.967Z
|
||||||
|
---
|
||||||
|
|
||||||
|
# Session Ledger — 2026-08-04
|
||||||
|
|
||||||
|
## Returns
|
||||||
|
|
||||||
|
- **2026-08-04T08:25 CEST — held the literal question rather than answering it.** The inherited question (`coherence_evaluated > 0` after replay) is one cheap query away, and I was in the DB already. Did not run it: the replay is at ~4%, so the query would answer a different question than the one inherited. Naming the temptation because §3's *completion-is-a-tripwire* shape was present — the cheapness of a check is not a reason to collapse the thread it belongs to.
|
||||||
|
- **2026-08-04T08:25 CEST — refused the wrap's rate projection instead of restating it.** The wrap left "~11 nodes/min, roughly 30 hours." Substrate said 778 nodes at 11.6 h uptime = 1.1/min lifetime. Rather than report either number, took the hourly histogram: 121 → 376 → ten hours at 5–23/hr → 172 in 25 min. The overnight collapse tracks machine sleep; the awake rate is ~7/min. Reported ≈45 h of *awake* time, explicitly on the wrap's own nodes-as-proxy. (Guards against §3 frame-inheritance: the wrap's projection was measured over a 15-minute awake window and does not extend to a wall-clock overnight.)
|
||||||
|
|
||||||
|
## What held
|
||||||
|
|
||||||
|
- **Substrate-check on an "outstanding" item.** Before listing `governance-drift-check.py` 3-of-5-inert as owed-but-unfiled, grepped `~/dotfiles/PENDING.md`: hits are older drift-check items, none is the inert-families finding. Verified against substrate, not against the wrap's memory of it.
|
||||||
|
|
||||||
|
## Open horizons
|
||||||
|
|
||||||
|
- **The replay's real completion condition is unknown.** ~19,925 events was the wrap's figure; nodes-as-proxy has not been shown to be 1:1 with events. Before any "the replay completed" claim, establish what the terminal node count should be.
|
||||||
|
- **`governance-drift-check.py` 3-of-5 families inert** — `[HARDENING]`, verified unfiled in PENDING.
|
||||||
|
- **Verification-ladder entries have no IDs**, so firing history is unrecordable by construction (same shape, one level up).
|
||||||
|
- **Today's named work**: what transfers CapableMind/BMF → library/engine; item 1 is census 01 against chamber/engine tooling (the census's own declared scope gap).
|
||||||
|
|
||||||
|
## Confidence to recalibrate
|
||||||
|
|
||||||
|
- ~~**Machine-sleep as the cause of the overnight collapse is inferred, not verified.** Confidence ~0.8.~~ **SUPERSEDED 2026-08-04T09:05 — REFUTED.** Steward supplied the falsifier (caffeine running). `pmset -g log`, domain-anchored and positive-controlled (the filter finds 450 `Sleep` / 453 `Wake` events historically), shows the **last sleep/wake event was 2026-08-03 08:54 CEST** — the machine has been continuously awake for 24 h. The 0.8 was too high for a claim resting entirely on a *shape* in a histogram with an available one-command check I did not run before reporting. **Return: I named the inference and its test, then reported the inference anyway rather than spending the one command.** Naming a check is not running it.
|
||||||
|
- **Corrected substrate, superseding the wake briefing.** The replay's denominator is **39,089 events**, not ~19,925; **nodes are not the progress metric** (6,468 nodes ≠ progress). Real position 22,300/39,089 = **57%** at 06:59 UTC, not ≈4%. Both wake numbers were wrong — wrong numerator *and* wrong denominator — because I proxied progress by a table I could count instead of reading the log the process was already writing. §3 *claim-from-derived-form when the substrate is checkable*.
|
||||||
|
- **Overnight cadence is metronomic: 100 events per ~51.5 min, eight consecutive intervals within 90 s of each other (51:29, 51:10, 51:30, 51:52, 51:54, 52:03, 51:33, 50:34) = ~31 s/event**, against `PIPELINE_TIMEOUT_MS` = 30,000 ms at `src/modules/entity/index.ts:102`. Every event paying the entity-pipeline timeout in full, serially, for nine hours. Regularity that tight is a timeout, not a load curve.
|
||||||
|
- **Unexplained and load-bearing: what flipped the regime at 06:24 UTC.** Rate went ~0.032 → 1.4–2.3 events/s (~50–70×) at the minute I made the first `/health` request in ten hours. Coincidence recorded as a coincidence; causation unestablished and I have no non-destructive test for it while the fast regime holds. **ETA spans two orders of magnitude on which regime governs: ~3 h fast, ~145 h slow.**
|
||||||
|
- Yesterday's standing recalibration carries: *knowing the name of a failure class confers no immunity to it* — seven uncontrolled instruments shipped on the day whose subject was uncontrolled instruments.
|
||||||
|
|
||||||
|
- **Entity-pipeline-as-the-brake: HYPOTHESIS REFUTED by a windowed count.** The 31 s/event cadence matched `PIPELINE_TIMEOUT_MS` so exactly that I nearly reported it as the cause. Anchoring log line numbers to timestamps and counting gave **0 entity timeouts in the overnight window** (lines 187892–188337, 20:05→04:44) against 2 in a 35-min morning window. A number that matches a constant is not evidence the constant fired. Second refuted inference in one morning; both died to a count.
|
||||||
|
- **The drain-gating mechanism is arithmetic-consistent, not call-path-traced.** 16 events/lease × ~8.2 min/cycle → 51.3 min per 100 events; observed 51.5 min (0.4%). Tight, but it is a coincidence-of-arithmetic argument. **Untraced gap: I did not verify that replay per-event work actually routes through `workQueue`.** Free falsifier available — leave the machine untouched 10 min and watch the cadence decay to the 8-minute orbit.
|
||||||
|
|
||||||
|
## Authorization moves
|
||||||
|
|
||||||
|
- **Owed, not filed:** `[HARDENING]` — the idle ladder's bottom half is unreachable. 2,393 transitions in the whole log, exactly two shapes (`active→warm` 1,197 / `warm→active` 1,196), **zero** to `cool` or `deep`, **zero** `idle_only` deferrable drains ever. `idle-state-machine.ts:204` — warm has no time-based exit; the only warm→cool path is `onAgentDisconnect`. Background training and consolidation have therefore never run on this instance. **Sixth instance of the governor-exists-and-never-engages class**, and the first found by looking for it rather than by tripping over it.
|
||||||
|
- **Owed, not filed:** `[FIX]` candidate — `BM_CPU_PAUSE_THRESHOLD` is unset in `~/.capablemind/env`, so the default 0.65 governs. The repo's own `CLAUDE.md` documents this as a starvation loop on Ollama-saturated hosts with the fix (`=1.0`) named. Documented, and not applied. 497 `replay paused for resource pressure` in the log.
|
||||||
|
|
||||||
|
- **FILED:** PENDING-92 `[HARDENING]` (idle ladder bottom half unreachable — spec §9A.1 divergence, sixth instance of the 08-03 class) · PENDING-93 `[PROPOSAL]` (`getChainsContainingSeq` / `event_seqs` normalisation — Seb's schema call).
|
||||||
|
- **APPLIED as `[FIX]`:** `BM_CPU_PAUSE_THRESHOLD=1.0` → `~/.capablemind/env`. Config-only, reversible by deleting the line, **inert until restart**. Scope stated honestly in the file itself: 6 pauses across 8.5 overnight hours, so this was never the main brake.
|
||||||
|
- **BUILT, NOT MERGED, NOT ENABLED:** branch `fix/idle-ladder-service-mode-cool-descent` (`d5e1e79`, `430cb5f`). Flag-off by default; 42/42 idle tests green; `tsc --noEmit` clean; full suite 4,003 tests. Authority: co-author build-on-a-branch (steward 2026-05-28). Merge + enable await steward authorization and Seb PR review per Constitutional Constraint #3. **Did not treat "close the three things" as blanket authorization** — the blast radius (first-ever run of a dormant subsystem) was established after the instruction was given.
|
||||||
|
|
||||||
|
## Returns (later)
|
||||||
|
|
||||||
|
- **Caught my own instrument reproducing the flaw it was built to escape.** `events_per_min` derived from a counter written once per 100 events read 0.0 across 24 consecutive samples while nodes advanced 169 and CPU ran 10–69%. The reading was *honest* and *illegible* — a frozen counter and zero throughput are different claims and nothing distinguished them. Fixed with `events_stale_seconds`. Third time today the lesson landed: **knowing the class confers no immunity**; I wrote the sampler *because* telemetry was too coarse and then derived a rate from the coarse source.
|
||||||
|
- **Did not claim a clean suite on a re-run.** First full run: 2 failures in `temporal.test.ts`; second full run at the same commit: 267/267. Rather than banking the pass, checked the file in isolation 3× (50/50 each) and confirmed the diff touches nothing temporal. Reported as **flaky under full-suite load**, not as green — the machine is saturated by the replay, which is itself the likely cause and worth watching.
|
||||||
|
|
||||||
|
## Open observation — RESOLVED, and it was arithmetic
|
||||||
|
|
||||||
|
- ~~**Nodes advance while the replay's event counter is frozen.**~~ **RESOLVED 2026-08-04T12:28 — not a mystery.** 593 nodes against 300 events = ~2 nodes/event; at 2.32 events/min the 100-event log granularity puts ~53 min between progress lines. Fully accounted. **I raised it as strange and it was division.** Recording the retraction because an unmarked "open observation" is a live claim, and the restraint that kept me from inventing a mechanism for it was the only thing that worked here.
|
||||||
|
|
||||||
|
## The fourth refutation, and the one that mattered
|
||||||
|
|
||||||
|
- **Ollama-saturation as the CPU-pause rationale: REFUTED.** I applied `BM_CPU_PAUSE_THRESHOLD=1.0` as a `[FIX]` in the morning on the assumption the governor was firing against a saturated host. Measured at 12:30: load 3.04, Ollama 1.2%, BMF 0.0%, top consumer Claude Code at 20%, and 3 pauses in 3 hours. **Fourth mechanism proposed and refuted in one day.** The fix stays (documented, reversible) but I undermined its own rationale and said so.
|
||||||
|
- **The generative error, named:** in a system with dozens of constants and a slow rate, coincidences are abundant, and I will always find one. 31 s/event ≈ 30,000 ms timeout. 8-min orbit ≈ 16-event lease. Both arithmetic, both wrong. **The pattern is not bad luck; it is a method that cannot fail to produce candidates.**
|
||||||
|
- **What actually found it: killing the thing and reading what it left behind.** Not profiling, not another instrument. `module_cursors`, read from a backup taken after clean shutdown, showed nine modules at ~24,000 and two at zero — and `minCursor = Math.min(...)` did the rest. Four months of profiling a running process could not see a value that is only legible at rest.
|
||||||
|
|
||||||
|
## Authorization moves (later)
|
||||||
|
|
||||||
|
- **FILED:** PENDING-94 `[ESCALATE]` — the resume floor. Tagged ESCALATE because `~/CLAUDE.md` names **cursor persistence** on the escalate-unconditionally list; the tag was determined by the constraint, not by my sense of severity.
|
||||||
|
- **AUTHORIZED BY STEWARD, EXECUTED:** killed the replay. Verified the premise first (2.32/min over 216 min → 4.9 days, same order as the 6-day figure the authorization rested on), stopped the service *before* copying so the SQLite snapshot could not tear, then verified the copy (391/391 files, 15/15 `quick_check: ok`).
|
||||||
|
- **PUSHED:** `CapableMind-AI@ad285df` — note to Seb, self-contained, correcting yesterday's note in its own first line. Every one of its five reproduction commands run verbatim before pushing.
|
||||||
|
|
||||||
|
## Sub-agent dialogues
|
||||||
|
|
||||||
|
## Bypasses
|
||||||
Executable
+358
@@ -0,0 +1,358 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
l1-replay-sampler.py — external, read-only progress sampler for a BMF replay.
|
||||||
|
|
||||||
|
WHY THIS EXISTS
|
||||||
|
---------------
|
||||||
|
The replay logs progress once per 100 events. In the slow regime that is one
|
||||||
|
data point every ~51 minutes — telemetry coarser than the failure mode it is
|
||||||
|
meant to describe. On 2026-08-04 three separate mechanism hypotheses (machine
|
||||||
|
sleep, entity-pipeline timeout, work-queue drain gating) each survived hours of
|
||||||
|
diagnosis because nothing at that resolution could contradict them. All three
|
||||||
|
were wrong. This samples the substrate directly, every 30s, and writes a curve.
|
||||||
|
|
||||||
|
It answers, continuously and without being asked:
|
||||||
|
- is the replay advancing, and at what instantaneous rate
|
||||||
|
- edges-per-node and chains-per-node as the graph densifies
|
||||||
|
- coherence_evaluated — the open question of whether that governor ever runs
|
||||||
|
- which idle state the process is in when each sample is taken
|
||||||
|
|
||||||
|
SAFETY
|
||||||
|
------
|
||||||
|
Read-only by discipline: only SELECTs are issued, nothing is restarted, no
|
||||||
|
config is touched. The DB is opened read-write rather than with mode=ro
|
||||||
|
because a mode=ro open of this instance FAILS — SQLite must be able to create
|
||||||
|
the -shm file to read a WAL database, and mode=ro cannot. (Earned 2026-08-03;
|
||||||
|
every read that day ran against a backup for this reason.) WAL readers do not
|
||||||
|
block the writer, so a 30s poll is negligible against a process writing
|
||||||
|
continuously.
|
||||||
|
|
||||||
|
INSTRUMENT DISCIPLINE
|
||||||
|
---------------------
|
||||||
|
Every source carries a same-run positive control. An absence is not evidence
|
||||||
|
until the instrument is shown capable of detecting presence — so if any source
|
||||||
|
cannot be read, or reads as structurally implausible, the sampler prints
|
||||||
|
INSTRUMENT NOT VERIFIED and refuses to start rather than emitting a clean-
|
||||||
|
looking stream of zeros. A sampler that silently reports nothing is exactly the
|
||||||
|
failure class it was built to investigate.
|
||||||
|
|
||||||
|
USAGE
|
||||||
|
-----
|
||||||
|
l1-replay-sampler.py --selftest run the controls, print, exit
|
||||||
|
l1-replay-sampler.py sample every 30s until stopped
|
||||||
|
l1-replay-sampler.py --interval 60 sample every 60s
|
||||||
|
l1-replay-sampler.py --once emit exactly one sample and exit
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
INSTANCE = os.environ.get("BM_INSTANCE_NAME", "mindfabric-00")
|
||||||
|
DB_PATH = Path.home() / ".capablemind" / "data" / INSTANCE / "sqlite" / "temporal.sqlite3"
|
||||||
|
LOG_PATH = Path.home() / ".capablemind" / "logs" / "bmf.stderr.log"
|
||||||
|
OUT_PATH = Path.home() / ".capablemind" / "diagnostics" / f"replay-samples-{INSTANCE}.jsonl"
|
||||||
|
|
||||||
|
TAIL_BYTES = 262_144 # 256 KB of log tail — enough to hold a progress line even when sparse
|
||||||
|
|
||||||
|
PROGRESS_RE = re.compile(r"Phase (\d) replay: (\d+)/(\d+) events")
|
||||||
|
IDLE_RE = re.compile(r"Idle state transition: (\w+) → (\w+) \(([^)]*)\)")
|
||||||
|
PAUSE_RE = re.compile(r"Phase \d replay paused for resource pressure")
|
||||||
|
|
||||||
|
|
||||||
|
def now_iso() -> str:
|
||||||
|
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# sources
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def read_db() -> dict:
|
||||||
|
"""Counts from the temporal store. SELECTs only."""
|
||||||
|
conn = sqlite3.connect(f"file:{DB_PATH}", uri=True, timeout=10.0)
|
||||||
|
try:
|
||||||
|
cur = conn.cursor()
|
||||||
|
out = {}
|
||||||
|
for key, sql in (
|
||||||
|
("nodes", "SELECT count(*) FROM temporal_node"),
|
||||||
|
("edges", "SELECT count(*) FROM caused"),
|
||||||
|
("chains", "SELECT count(*) FROM causal_chain"),
|
||||||
|
("coherence_evaluated", "SELECT count(*) FROM causal_chain WHERE coherence_evaluated = 1"),
|
||||||
|
):
|
||||||
|
out[key] = cur.execute(sql).fetchone()[0]
|
||||||
|
return out
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def read_log_tail() -> dict:
|
||||||
|
"""Last replay position, last idle transition, and pause count in the tail window."""
|
||||||
|
size = LOG_PATH.stat().st_size
|
||||||
|
with LOG_PATH.open("rb") as fh:
|
||||||
|
fh.seek(max(0, size - TAIL_BYTES))
|
||||||
|
tail = fh.read().decode("utf-8", errors="replace")
|
||||||
|
|
||||||
|
out: dict = {
|
||||||
|
"phase": None,
|
||||||
|
"events_done": None,
|
||||||
|
"events_total": None,
|
||||||
|
"idle_state": None,
|
||||||
|
"idle_reason": None,
|
||||||
|
"pauses_in_tail": len(PAUSE_RE.findall(tail)),
|
||||||
|
"log_bytes": size,
|
||||||
|
}
|
||||||
|
|
||||||
|
progress = PROGRESS_RE.findall(tail)
|
||||||
|
if progress:
|
||||||
|
phase, done, total = progress[-1]
|
||||||
|
out["phase"] = int(phase)
|
||||||
|
out["events_done"] = int(done)
|
||||||
|
out["events_total"] = int(total)
|
||||||
|
|
||||||
|
idle = IDLE_RE.findall(tail)
|
||||||
|
if idle:
|
||||||
|
_frm, to, reason = idle[-1]
|
||||||
|
out["idle_state"] = to
|
||||||
|
out["idle_reason"] = reason
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def read_process() -> dict:
|
||||||
|
"""CPU seconds consumed — distinguishes 'working slowly' from 'not working'."""
|
||||||
|
try:
|
||||||
|
pids = subprocess.run(
|
||||||
|
["pgrep", "-f", "BetterMemories.io/dist/index.js"],
|
||||||
|
capture_output=True, text=True, timeout=10,
|
||||||
|
).stdout.split()
|
||||||
|
if not pids:
|
||||||
|
return {"pid": None, "cpu_seconds": None, "alive": False}
|
||||||
|
pid = int(pids[0])
|
||||||
|
raw = subprocess.run(
|
||||||
|
["ps", "-o", "time=", "-p", str(pid)],
|
||||||
|
capture_output=True, text=True, timeout=10,
|
||||||
|
).stdout.strip()
|
||||||
|
# ps time format: [[dd-]hh:]mm:ss
|
||||||
|
days, _, rest = raw.partition("-")
|
||||||
|
if not rest:
|
||||||
|
days, rest = "0", days
|
||||||
|
parts = [float(p) for p in rest.split(":")]
|
||||||
|
while len(parts) < 3:
|
||||||
|
parts.insert(0, 0.0)
|
||||||
|
secs = int(days) * 86400 + parts[0] * 3600 + parts[1] * 60 + parts[2]
|
||||||
|
return {"pid": pid, "cpu_seconds": round(secs, 2), "alive": True}
|
||||||
|
except Exception as exc: # noqa: BLE001 — degradation must be visible, not silent
|
||||||
|
return {"pid": None, "cpu_seconds": None, "alive": False, "error": str(exc)}
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# positive controls
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def selftest() -> tuple[bool, list[str]]:
|
||||||
|
"""
|
||||||
|
Each control must demonstrate the instrument can SEE something, not merely
|
||||||
|
that it did not crash. Returns (ok, lines).
|
||||||
|
"""
|
||||||
|
lines: list[str] = []
|
||||||
|
ok = True
|
||||||
|
|
||||||
|
# Control 1 — the DB exists, opens, and the four counters return integers.
|
||||||
|
try:
|
||||||
|
db = read_db()
|
||||||
|
if not all(isinstance(v, int) for v in db.values()):
|
||||||
|
ok = False
|
||||||
|
lines.append("FAIL db: a counter returned a non-integer")
|
||||||
|
elif db["nodes"] == 0 and db["edges"] == 0 and db["chains"] == 0:
|
||||||
|
ok = False
|
||||||
|
lines.append("FAIL db: all three counters are zero — cannot distinguish "
|
||||||
|
"'empty store' from 'reading the wrong database'")
|
||||||
|
else:
|
||||||
|
lines.append(f"pass db: nodes={db['nodes']} edges={db['edges']} "
|
||||||
|
f"chains={db['chains']} coherence_evaluated={db['coherence_evaluated']}")
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
ok = False
|
||||||
|
lines.append(f"FAIL db: {exc}")
|
||||||
|
|
||||||
|
# Control 2 — the log is readable AND the progress regex actually matches.
|
||||||
|
# A regex that matches nothing would report 'no progress' forever.
|
||||||
|
try:
|
||||||
|
log = read_log_tail()
|
||||||
|
if log["events_done"] is None:
|
||||||
|
ok = False
|
||||||
|
lines.append("FAIL log: no 'Phase N replay: X/Y events' line in the last "
|
||||||
|
f"{TAIL_BYTES // 1024} KB — the progress parser is unverified")
|
||||||
|
else:
|
||||||
|
lines.append(f"pass log: phase={log['phase']} "
|
||||||
|
f"events={log['events_done']}/{log['events_total']}")
|
||||||
|
if log["idle_state"] is None:
|
||||||
|
lines.append("warn log: no idle transition in the tail window "
|
||||||
|
"(idle_state will read null; not fatal)")
|
||||||
|
else:
|
||||||
|
lines.append(f"pass log: idle_state={log['idle_state']} ({log['idle_reason']})")
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
ok = False
|
||||||
|
lines.append(f"FAIL log: {exc}")
|
||||||
|
|
||||||
|
# Control 3 — the process is locatable and reports non-zero CPU.
|
||||||
|
proc = read_process()
|
||||||
|
if not proc["alive"]:
|
||||||
|
ok = False
|
||||||
|
lines.append(f"FAIL proc: BMF process not found ({proc.get('error', 'no match')})")
|
||||||
|
elif not proc["cpu_seconds"]:
|
||||||
|
ok = False
|
||||||
|
lines.append("FAIL proc: cpu_seconds parsed as zero — the ps parser is unverified")
|
||||||
|
else:
|
||||||
|
lines.append(f"pass proc: pid={proc['pid']} cpu_seconds={proc['cpu_seconds']}")
|
||||||
|
|
||||||
|
# Control 4 — the output path is writable.
|
||||||
|
try:
|
||||||
|
OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with OUT_PATH.open("a"):
|
||||||
|
pass
|
||||||
|
lines.append(f"pass out: {OUT_PATH} writable")
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
ok = False
|
||||||
|
lines.append(f"FAIL out: {exc}")
|
||||||
|
|
||||||
|
return ok, lines
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# sampling
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def sample(prev: dict | None, events_changed_at: float | None = None) -> dict:
|
||||||
|
rec: dict = {"ts": now_iso()}
|
||||||
|
try:
|
||||||
|
rec.update(read_db())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
rec["db_error"] = str(exc)
|
||||||
|
try:
|
||||||
|
rec.update(read_log_tail())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
rec["log_error"] = str(exc)
|
||||||
|
rec.update(read_process())
|
||||||
|
|
||||||
|
# Derived ratios — the densification curve, which is the thing that decides
|
||||||
|
# whether the rebuild bought a fix or only a constant factor.
|
||||||
|
if rec.get("nodes"):
|
||||||
|
rec["edges_per_node"] = round(rec.get("edges", 0) / rec["nodes"], 2)
|
||||||
|
rec["chains_per_node"] = round(rec.get("chains", 0) / rec["nodes"], 2)
|
||||||
|
|
||||||
|
# Instantaneous rates against the previous sample.
|
||||||
|
if prev and prev.get("ts"):
|
||||||
|
dt = (datetime.fromisoformat(rec["ts"]) - datetime.fromisoformat(prev["ts"])).total_seconds()
|
||||||
|
if dt > 0:
|
||||||
|
if rec.get("nodes") is not None and prev.get("nodes") is not None:
|
||||||
|
rec["nodes_per_min"] = round((rec["nodes"] - prev["nodes"]) * 60 / dt, 2)
|
||||||
|
if rec.get("events_done") is not None and prev.get("events_done") is not None:
|
||||||
|
rec["events_per_min"] = round((rec["events_done"] - prev["events_done"]) * 60 / dt, 2)
|
||||||
|
if rec.get("cpu_seconds") is not None and prev.get("cpu_seconds") is not None:
|
||||||
|
rec["cpu_percent"] = round((rec["cpu_seconds"] - prev["cpu_seconds"]) * 100 / dt, 1)
|
||||||
|
|
||||||
|
# events_per_min is derived from a counter the replay only writes every 100
|
||||||
|
# events. At 12 events/min that is one update per ~8 minutes, so the rate
|
||||||
|
# reads 0.0 for most samples and then spikes — the same coarse-telemetry
|
||||||
|
# flaw this sampler exists to escape, reproduced one level in. Publish how
|
||||||
|
# long the counter has been FROZEN so a flat reading is legible as stale
|
||||||
|
# rather than as zero throughput, and treat nodes_per_min as the fine signal.
|
||||||
|
if events_changed_at is not None:
|
||||||
|
rec["events_stale_seconds"] = round(time.time() - events_changed_at, 1)
|
||||||
|
return rec
|
||||||
|
|
||||||
|
|
||||||
|
def seed_staleness() -> tuple[int | None, float | None]:
|
||||||
|
"""
|
||||||
|
Recover the events counter and when it last CHANGED from the existing
|
||||||
|
stream, so a restart of the sampler does not reset the staleness clock.
|
||||||
|
|
||||||
|
Without this, restarting the instrument makes a counter frozen for 33
|
||||||
|
minutes report stale=0.0s — the instrument's own restart erasing the very
|
||||||
|
signal it exists to publish. Observed 2026-08-04 on the first handoff
|
||||||
|
between two sampler processes.
|
||||||
|
"""
|
||||||
|
if not OUT_PATH.exists():
|
||||||
|
return None, None
|
||||||
|
try:
|
||||||
|
size = OUT_PATH.stat().st_size
|
||||||
|
with OUT_PATH.open("rb") as fh:
|
||||||
|
fh.seek(max(0, size - TAIL_BYTES))
|
||||||
|
lines = fh.read().decode("utf-8", errors="replace").splitlines()
|
||||||
|
rows = []
|
||||||
|
for line in lines:
|
||||||
|
try:
|
||||||
|
rows.append(json.loads(line))
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue # a partial first line from the seek, or a torn write
|
||||||
|
rows = [r for r in rows if r.get("events_done") is not None]
|
||||||
|
if not rows:
|
||||||
|
return None, None
|
||||||
|
current = rows[-1]["events_done"]
|
||||||
|
# Walk back to the first record still holding the current value.
|
||||||
|
changed_ts = rows[-1]["ts"]
|
||||||
|
for row in reversed(rows):
|
||||||
|
if row["events_done"] != current:
|
||||||
|
break
|
||||||
|
changed_ts = row["ts"]
|
||||||
|
return current, datetime.fromisoformat(changed_ts).timestamp()
|
||||||
|
except Exception: # noqa: BLE001 — seeding is best-effort; never block sampling
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("--interval", type=float, default=30.0, help="seconds between samples (default 30)")
|
||||||
|
ap.add_argument("--once", action="store_true", help="emit one sample and exit")
|
||||||
|
ap.add_argument("--selftest", action="store_true", help="run positive controls and exit")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
ok, lines = selftest()
|
||||||
|
for line in lines:
|
||||||
|
print(line, file=sys.stderr)
|
||||||
|
if not ok:
|
||||||
|
print("\nINSTRUMENT NOT VERIFIED — refusing to sample. A stream of zeros from a "
|
||||||
|
"blind instrument is worse than no stream at all.", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
if args.selftest:
|
||||||
|
print("\ninstrument verified", file=sys.stderr)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
print(f"sampling every {args.interval:g}s → {OUT_PATH}", file=sys.stderr)
|
||||||
|
|
||||||
|
prev: dict | None = None
|
||||||
|
last_events, events_changed_at = seed_staleness()
|
||||||
|
if last_events is not None:
|
||||||
|
held = time.time() - (events_changed_at or time.time())
|
||||||
|
print(f"resumed staleness: events={last_events} unchanged for {held:.0f}s", file=sys.stderr)
|
||||||
|
while True:
|
||||||
|
rec = sample(prev, events_changed_at)
|
||||||
|
if rec.get("events_done") is not None and rec["events_done"] != last_events:
|
||||||
|
last_events = rec["events_done"]
|
||||||
|
events_changed_at = time.time()
|
||||||
|
rec["events_stale_seconds"] = 0.0
|
||||||
|
with OUT_PATH.open("a") as fh:
|
||||||
|
fh.write(json.dumps(rec) + "\n")
|
||||||
|
prev = rec
|
||||||
|
if args.once:
|
||||||
|
print(json.dumps(rec, indent=2), file=sys.stderr)
|
||||||
|
return 0
|
||||||
|
time.sleep(args.interval)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
sys.exit(main())
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
sys.exit(130)
|
||||||
Reference in New Issue
Block a user