dotfiles/claude/memory/feedback-bmf-health-vs-recall.md

name, description, type, originSessionId, permalink

name	description	type	originSessionId	permalink
BMF module health does not measure recall correctness	Modules reporting status=ready / system_status=healthy in BMF measures pipeline liveness only — not whether recall produces correct results. Verify recall directly. Never trust the health endpoint as a proxy for "the system works."	feedback	a3ddbf34-da48-4129-acba-658f2f869783	claude-memory/feedback-bmf-health-vs-recall

Module health is decoupled from recall correctness

When evaluating BMF state — for any session, any audit, any "is it working" question — never use module health (status: ready, system_status: healthy) as evidence that recall produces correct results. The two are structurally decoupled.

Why: The health endpoint measures pipeline liveness — subscription up, storage reachable, cursor advancing, error_count not catastrophic. None of those signals address the question "if I ask 'Who is Kai?', do I get vault content back?" Verified empirically 2026-04-19: all 11 modules reported ready with system_status: healthy, and 8 of 15 designed recall queries returned empty (including known names like Lune, Kai, Nuria); 6 returned junk (temporal-stats blobs or session-hook-pollution); only 1 returned honestly empty. The system reported healthy throughout.

This is a violation of the honest-degradation invariant (bettermemories/CLAUDE.md invariant: "Honest degradation — the system must report its own limits. Silent failures are architectural violations") at design level, not at instance level. There is currently no pipeline from "recall is emitting noise" to "module health should flip." That gap is itself an architectural finding bigger than any single bug it produces.

The deeper pattern (surfaced 2026-04-19 audit-bundle architect-review)

One level underneath the symptom: BMF's honest-degradation contract applies only to the write path; the read path has no equivalent scaffolding. Write-path modules report cursor, error_count, last_processed_at — measurable observables. Read-path subsystems (query-planner, query-router, hybridSearch, temporal fallthrough, working-memory injection) silently produce empty or junk results with no diagnostic surface — no counter, no error signal, no "why empty" breadcrumb. The slow-query log at vector/queries.ts:291 only fires on searchMs > 200, so fast 0-return queries are invisible.

H1–H4 from the 2026-04-19 audit are not four independent bugs; they are four confirmations of an absent contract on the read path. Patching them one-by-one fixes symptoms; naming the pattern lets the read path get the observability discipline the write path already has.

How to apply this deeper framing:

• When auditing any new behavior in BMF, ask first whether it's on the write path or the read path.
• On the write path, the honest-degradation discipline is already in place — bugs there are usually local.
• On the read path, assume the bug you find is one instance of a broader silence. Look for siblings.
• When proposing fixes, consider whether the fix should be the instance (e.g., H1's #126 guard) or the contract (read-path observability layer). Often both — ship the instance, file the contract as architectural follow-on.

How to apply

For any audit / diagnostic / status check:

• Run actual recall queries against known vault content (a name, a thematic term, a recent event).
• Test with min_confidence: 0 AND with default — the gap between them reveals whether content exists in storage but is being filtered.
• Test with explicit modules: ["vector"] — vector should return semantic hits even if entity is degraded.
• Be alert to two synthetic-fallback patterns that masquerade as content:
  1. Working-memory injection: results with content like "Recent event: events.agent.tool.result (...)" and confidence 0.8 — these come from query-router.ts:268-290 injecting the agent's OWN current tool calls, not vault content.
  2. Temporal stats fallthrough: results with content like {"total_nodes":N,"total_edges":M,...} — the temporal module returning its own graph stats regardless of query intent.
• If either fallback pattern appears, recall is producing junk; the module health report is non-load-bearing.

For session reporting / steward updates:

• Do NOT report "recall operational" based on a single high-confidence result without a baseline across multiple query categories.
• Do NOT report "system healthy" based on health endpoint without verifying recall.
• Do NOT conflate "ingestion completed" with "the system can use what was ingested" — these are structurally separate (substrate vs interpretation).

For future BMF work:

• Until BMF has a recall-correctness signal in module health (or system_status), the operator (steward, executor, eventually end-user) is the only validator. That's a structural problem worth surfacing as a [PROPOSAL] to Seb when the moment fits.

Background — when this was learned

Steward verbatim, after the recall baseline test revealed the decoupling:

"So my joy at reaching the milestone was premature I now understand... I can accept that. Now to make this work."

Yesterday's milestone (2026-04-18 evening): connector completed 7,220 items, 14,355 entities extracted, 0 failed, all modules ready, single-query recall returned confidence 1.0 → steward called it "the foundation." That reading was true about substrate-bytes-flowed but premature about recall-works. The foundation in the deeper sense (the system can be used as cognitive memory) requires recall correctness, which today's audit shows is not yet present.