David F GliddenandClaude Opus 4.8
3f9a89b00c
Basic Memory v0.21.6 first sync over the live memory dir (steward-authorized live-dir trial, Option A 2026-06-06): adds permalink: to frontmatter, refolds long YAML description lines, strips final newlines. Bodies untouched — verified via full diff classification. From this commit forward, any diff in claude/memory shows only what Basic Memory or the session writes. Trial design: MemPalace untouched as incumbent; git status check on this dir at every wrap; end-of-day evaluation (recall quality, sync robustness, rebuild-from-files, malformed-file behavior). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
50 lines
3.4 KiB
Markdown
50 lines
3.4 KiB
Markdown
---
|
|
name: BMF summarizer-connector coupling
|
|
description: Factory's maybeGenerateSummary is silently coupled to the local_files
|
|
observation shape; most connectors fall through. Deferred refactor target.
|
|
type: project
|
|
originSessionId: 1c1bf37d-b1c3-4a62-9355-613c50f4c028
|
|
permalink: claude-memory/bmf-summarizer-contract
|
|
---
|
|
|
|
# BMF summarizer-connector coupling (deferred refactor)
|
|
|
|
**Surfaced:** 2026-04-14, during the SQLite+LanceDB recall unblock session.
|
|
|
|
`src/factory/factory.ts::maybeGenerateSummary` implicitly requires the local_files observation shape:
|
|
- `source.item_type ∈ {'file','image','passage'}`
|
|
- `payload.metadata.file_path` (or `source.item_id`)
|
|
- `payload.metadata.mime_type` (unless item_type is `'file'` or `'image'`)
|
|
- `payload.metadata.size_bytes`
|
|
- `source.content_hash`
|
|
|
|
When `documentSummarizer.isAvailable()` is true, `processFactoryBatch` previously set `skipDispatch: true` uniformly for the whole batch. Any observation that didn't match the summarizer's entry guard was written to logchain and then silently dropped — never dispatched, never summarized. This was the actual cause of zero embeddings after 2,457 Obsidian notes ingested.
|
|
|
|
**Audit of `mime_type` usage across connectors (2026-04-14):**
|
|
- Sets it: local_files, google, obsidian (post-fix).
|
|
- Does NOT: claude_transcripts, slack, github, apple_calendar, apple_contacts, imessage, git, file_import.
|
|
|
|
Every connector in the second group drops content silently when a summarizer is available — until the partition fix landed.
|
|
|
|
## Why: fixes applied in this session
|
|
|
|
- **Structural safety net** (`factory.ts::processFactoryBatch`): partition the batch by a summary-eligibility predicate mirroring `maybeGenerateSummary`'s entry guard. Summarizable → `skipDispatch: true`; pass-through → `skipDispatch: false`. Guarantees every content type reaches modules regardless of summarizer shape.
|
|
- **Notes first-class** (`connectors/obsidian.ts`, `factory.ts::maybeGenerateSummary`): emit `mime_type: 'text/markdown'` + `file_path`; extend summarizer `'file'` branch to accept `'note'`. Notes get the quality path (summary, teacher enrichment eligible).
|
|
- **Provenance** (`factory.ts::generateAndObserveSummary`): `connector_id` and `factory:*` tag now use `job.connector_id` instead of hardcoded `'local_files'`.
|
|
|
|
## How to apply: the remaining `[PROPOSAL]`
|
|
|
|
**Not** "add `mime_type` everywhere." Most connectors (git, apple_contacts, apple_calendar, slack/imessage messages, claude_transcripts per-turn, short github comments) are correctly pass-through and should NOT be summarized. The right generalization is an explicit contract:
|
|
|
|
- Add `summarizable: boolean` to `ConnectorObservation` (or equivalent) — chosen per-connector.
|
|
- Dispatch partition uses the flag directly rather than inferring via item_type/mime_type.
|
|
- Summarizer accepts a generic-text shape (title + content + content_hash), not local_files-shaped metadata.
|
|
- Fail-loud telemetry: a batch with zero summaries AND zero pass-through dispatches should alarm, not silently succeed.
|
|
|
|
Touches connector contracts — deserves its own focused PR with Seb reviewing, not a tail-end addition to the storage-migration PR.
|
|
|
|
## When to revisit
|
|
|
|
- When opening the next L1 PR after the storage-migration one merges.
|
|
- Before onboarding any new text-bearing connector (the wrong default will keep catching people).
|
|
- If any recall quality investigation points at content-not-dispatched as a suspect — this is the mechanism. |