--- name: mempalace-chunk-by-paragraph-oversized-chunk-bug-diagnostic-record-2026-05-16 description: 'Specific mempalace bug identified the night of 2026-05-16 after a mine attempt of ~/.claude/projects/ failed with 120 GiB attention buffer error. Root cause: _chunk_by_paragraph (convo_miner.py:184-186 + 176-182) does not enforce CHUNK_SIZE, letting paragraphs become single oversized chunks that crash the embedding model. Reproducer + line numbers + fix shape recorded; decision pending on whether to file upstream as GH issue.' metadata: node_type: memory type: project originSessionId: 8952fe77-1cf0-401d-9d88-01f9f0b13bcc permalink: claude-memory/project-mempalace-chunking-bug-2026-05-16 superseded_by: project-mempalace-winddown.md superseded_on: 2026-08-17 --- > **SUPERSEDED 2026-08-17.** Filed upstream as MemPalace/mempalace#1534 and left to upstream by steward direction. With the instrument retired this is no longer ours to track. > > Current record: **project-mempalace-winddown.md**. Kept for detail and provenance — do not read the > status below as live. # MemPalace `_chunk_by_paragraph` oversized-chunk bug **Discovered:** 2026-05-16 night **Status:** FILED upstream as MemPalace/mempalace#1534 (2026-05-17) — https://github.com/MemPalace/mempalace/issues/1534 **Follow-up evidence posted:** 2026-05-17 — https://github.com/MemPalace/mempalace/issues/1534#issuecomment-4469591465 (broader scope: both bug locations firing in practice; tool-results .txt files with 834 KB single lines hit the line-group fallback; convos-mode mining is effectively blocked on any real Claude Code corpus). Severity framed toward "high" in the follow-up. **Resolved-locally:** No (per steward direction: "not fix someone else's work"); waiting on upstream fix. Operations 2/3/4 of the four-op plan remain held until upstream ships. **Reproduction note (2026-05-17 morning):** Quarantining the two original-repro files (`e31838f5` + `8952fe77` → `.jsonl.QUARANTINE-2026-05-17`) and re-running the mine crashed again on the next non-skipped file in iteration order with identical 120 GiB upsert error. Restored files cleanly; palace state unchanged at 14,496 drawers. **Side effect:** during the brief quarantine window (~30 seconds), Claude Code recreated `8952fe77-...jsonl` with in-flight session content; the restore-via-mv overwrote that recreation. The few minutes of session-turn writes in that window are missing from the on-disk transcript (session memory file + this bug record capture the meaningful conclusions). Recoverable but worth knowing if future wake-up reads of 8952fe77.jsonl show a content gap. --- ## Summary `mempalace mine --mode convos` can crash with `RuntimeError: Invalid buffer size: 120.00 GiB in upsert.` when a session jsonl file contains content that gets routed through `_chunk_by_paragraph` (the fallback chunker) AND has a paragraph longer than what the embedding model's attention computation can hold. Today's reproducer is a Claude Code session jsonl containing a large pasted document (~30KB) as a single user turn — the paragraph chunker doesn't split it; embedding tries to compute attention on a single oversized chunk and runs out of memory. The bug is localized to **two specific code paths** in `convo_miner.py`. The peer chunker `_chunk_by_exchange` handles size correctly. Storage layer, embedding layer, MCP, hooks, search, KG — all working. The damage is bounded to this one path. --- ## Repro **Environment:** - mempalace local HEAD: `604df3b local: raise MAX_CHUNKS_PER_FILE 500→50_000 for full-text scholarly corpora` (note: MAX_CHUNKS_PER_FILE is in `miner.py`, not relevant to this bug; mentioned only for environment fingerprint) - Python 3.13, embedding model: xlm-roberta variant (391 weights loaded per startup banner) - macOS Apple Silicon, palace at `~/.mempalace/palace-memory` **Trigger:** ``` mempalace mine ~/.claude/projects/ --mode convos --wing claude-sessions --agent claude-code ``` **Observed:** - First file processed: traceback as below - Zero drawers added on the failed run - Palace state unchanged at 14,494 drawers (already populated by an earlier same-day successful mine that ran 16:47–17:47 before the trigger files appeared) **Traceback (top of stack):** ``` File "/Users/davidglidden/_Dev/mempalace/mempalace/convo_miner.py", line 489, in mine_convos drawers_added, room_delta, skipped = _file_chunks_locked( File "/Users/davidglidden/_Dev/mempalace/mempalace/convo_miner.py", line 370, in _file_chunks_locked collection.upsert( documents=batch_docs, … File ".../transformers/integrations/sdpa_attention.py", line 92, in sdpa_attention_forward attn_output = torch.nn.functional.scaled_dot_product_attention(...) RuntimeError: Invalid buffer size: 120.00 GiB in upsert. ``` Full failing-run log preserved at `/private/tmp/claude-501/-Users-davidglidden/8952fe77-1cf0-401d-9d88-01f9f0b13bcc/tasks/bm45temj3.output` (172 lines, the relevant lines are start banner + the upsert traceback). **The specific input that triggers it (2026-05-16):** Two Claude Code session jsonl files created today: - `~/.claude/projects/-Users-davidglidden/e31838f5-238e-446f-888d-7adab6eaf318.jsonl` — 2.88 MB, 460 lines, max line length 135,600 chars - `~/.claude/projects/-Users-davidglidden/8952fe77-1cf0-401d-9d88-01f9f0b13bcc.jsonl` — 2.10 MB, 618 lines, max line length 119,671 chars These contain large pasted content (an ADR document paste was ~30KB; Read tool results returned ~100KB markdown files multiple times) as single jsonl lines. **Dry-run chunk counts as a smoking-gun comparison:** | File | Source | Chunks | Avg chunk size | |---|---|---|---| | `a156edbb-...jsonl` | older Claude Code session, similar wall-clock size | 1352 | ~2 KB | | `e31838f5-...jsonl` | TODAY (20:31) | **170** | **~17 KB** | | `8952fe77-...jsonl` | TODAY (22:18) | **135** | **~15 KB** | The older file went through `_chunk_by_exchange` and produced safe 2KB chunks. The two new files went through `_chunk_by_paragraph` and produced massively oversized chunks. The `_chunk_by_exchange` vs `_chunk_by_paragraph` dispatch is decided by `quote_lines >= 3` check at `convo_miner.py:108` — whether the normalized content has ≥3 lines starting with `>`. Why the two new files don't meet that threshold needs verification but is likely a `normalize()` behavior difference for very-recent Claude Code jsonl format. --- ## Bug location `mempalace/convo_miner.py`: ### Bug 1 — paragraph chunking (lines 184–186) ```python def _chunk_by_paragraph(content: str) -> list: """Fallback: chunk by paragraph breaks.""" chunks = [] paragraphs = [p.strip() for p in content.split("\n\n") if p.strip()] # If no paragraph breaks and long content, chunk by line groups if len(paragraphs) <= 1 and content.count("\n") > 20: lines = content.split("\n") for i in range(0, len(lines), 25): group = "\n".join(lines[i : i + 25]).strip() if len(group) > MIN_CHUNK_SIZE: chunks.append({"content": group, "chunk_index": len(chunks)}) return chunks for para in paragraphs: if len(para) > MIN_CHUNK_SIZE: chunks.append({"content": para, "chunk_index": len(chunks)}) # ← BUG: no CHUNK_SIZE cap return chunks ``` A paragraph above `CHUNK_SIZE` (800 chars) is appended as a single oversized chunk. No splitting. ### Bug 2 — line-group fallback (lines 178–181) Same shape: groups 25 lines at a time, doesn't enforce `CHUNK_SIZE` on the resulting group. ### Contrast — the working path (lines 145-156, `_chunk_by_exchange`) ```python if len(content) > CHUNK_SIZE: first_part = content[:CHUNK_SIZE] if len(first_part.strip()) > MIN_CHUNK_SIZE: chunks.append({"content": first_part, "chunk_index": len(chunks)}) remainder = content[CHUNK_SIZE:] while remainder: part = remainder[:CHUNK_SIZE] remainder = remainder[CHUNK_SIZE:] if len(part.strip()) > MIN_CHUNK_SIZE: chunks.append({"content": part, "chunk_index": len(chunks)}) ``` This is the correct shape. The fix for `_chunk_by_paragraph` is to mirror this splitting loop in both bug locations. --- ## Suggested fix (sketch, not committed) In `_chunk_by_paragraph`, both paths need a size-bounded splitter. Suggested helper: ```python def _emit_bounded(chunks: list, content: str) -> None: """Append content as one or more chunks, none exceeding CHUNK_SIZE.""" while content: part = content[:CHUNK_SIZE] content = content[CHUNK_SIZE:] if len(part.strip()) > MIN_CHUNK_SIZE: chunks.append({"content": part, "chunk_index": len(chunks)}) ``` Then replace both bug sites: ```python # Line ~180 (line-group fallback): for i in range(0, len(lines), 25): group = "\n".join(lines[i : i + 25]).strip() _emit_bounded(chunks, group) # Line ~184 (paragraph loop): for para in paragraphs: _emit_bounded(chunks, para) ``` This makes `_chunk_by_paragraph` symmetric in size-discipline with `_chunk_by_exchange`. --- ## Honest accounting of what is known vs hypothesized ### Known with high confidence - `_chunk_by_paragraph` does not enforce `CHUNK_SIZE`. Code is verbatim above; the absence of splitting is visible. - The two new today-evening jsonl files produced anomalous chunk counts (170 and 135) on dry-run; older similar-size files produced 1000+ chunks. The ratio is the smoking gun. - The mine failed with a 120 GiB attention buffer error on the first non-skipped file's upsert. - `_chunk_by_exchange` handles size correctly (lines 145-156). - Earlier same-day mine (16:47–17:47) succeeded across ~245 files; only the two new files added after 17:47 are candidates for triggering the crash. ### Hypothesis (high confidence but not direct-verified) - The exact failing file is one of `e31838f5-...` or `8952fe77-...`. Dry-run chunk-counts strongly imply both would crash on embedding, but the live mine failed on the FIRST non-skipped file; whichever the iterator hit first is the one that crashed. Not directly checked. - The 120 GiB number comes from attention being O(seq_len²) and the embedding model trying to handle a 30K–50K-token-equivalent single chunk. The exact arithmetic depends on model + batch + tokenization specifics; the order-of-magnitude fits a 100KB-paragraph chunk. ### Open questions worth checking before filing 1. **Why don't the two new jsonl files trigger the `quote_lines >= 3` branch?** Are recent Claude Code jsonl files normalized differently? Could be a recent `normalize()` change, or just that today's sessions happen to lack `>`-prefixed quotes. Worth checking by inspecting `normalize()` output on the two files (the verification Python script tonight failed on a missing chromadb module in the system Python; needs to run inside mempalace's venv). 2. **Was the chunking bug present in earlier mempalace versions, or recent?** Worth a git-blame on `_chunk_by_paragraph`. If recent, more urgent. 3. **Does `--mode projects` (miner.py) have the same shape of bug in its chunker?** Operations 2/3/4 of the four-op plan would use projects mode; needs separate verification. 4. **Why did the earlier-today 16:47 mine succeed?** Probably because (a) it ran before the two new jsonl files appeared, and (b) older jsonl files all went through `_chunk_by_exchange`. Both verifiable from dry-run output. ### What this is NOT - NOT a storage corruption bug. The 14,494 drawers already in `palace-memory` are intact and searchable. - NOT a re-mining-duplicates bug. `file_already_mined` skips already-ingested files before any upsert; duplicate "already exists" errors are caught at upsert. (My first hypothesis tonight conflated these; corrected after steward push-back.) - NOT a `MAX_CHUNKS_PER_FILE` issue. That constant lives in `miner.py` (project files), not `convo_miner.py`. - NOT a chromadb / embedding-model installation bug. Embedding works for normal-sized chunks; only oversized ones crash. --- ## If filing as a GitHub issue **Title suggestion:** `convo_miner: _chunk_by_paragraph does not enforce CHUNK_SIZE, can produce single oversized chunks that crash embedding` **Body skeleton:** - Summary (one paragraph, the bug shape) - Reproducer (env + the dry-run chunk-count comparison) - Code locations (convo_miner.py:184-186, 176-182) - Suggested fix (the `_emit_bounded` helper, both call-sites) - What's affected: `mempalace mine --mode convos` against any directory containing jsonl files with large single-line content that doesn't trigger the `quote_lines >= 3` branch **Severity framing:** medium-high — the failure is loud (exception) rather than silent-corruption, but it stops a long-running operation cold and the user wastes the embedding setup time. With trust-cost factored in (a steward who has lost weeks to mining time), severity reads higher. **Repo to file against:** MemPalace/mempalace — same repo where #1526 (cascade + silent-empty-index) was filed earlier today. --- ## Trust-and-decision context The steward stated "I need to be able to trust mempalace" tonight after losing weeks to mining-related issues. This bug is specific, bounded, reproducible, and mechanically fixable. The damage from this particular bug is bounded to: - The chunker fallback path (one function, two lines each) - Sessions where normalized content lacks `>`-quote markers AND has paragraphs above CHUNK_SIZE The broader trust concern is mining-time-vs-failure-mode visibility: long mines that fail near the end (or, here, on the first non-skipped file after a setup) are expensive even when they fail loudly, and exit-0-via-`tee` masking of the underlying nonzero exit (a shell pattern, my error tonight, not mempalace's) compounds this. Decision recommendation for steward (offered, not made): - **File the GH issue** if you want this fixed upstream and shipped to other users. Bug is concrete; fix is small; report would be useful. - **Defer filing + fix locally** if you want immediate working ingestion for ops 2/3/4 without waiting on upstream merge. The `_emit_bounded` helper is ~5 minutes of work in a fresh head. - **Defer + accept current corpus as-is** if you want to use what's already in palace-memory and not push more ingestion until upstream ships a fix. Either way: don't retry `mempalace mine ~/.claude/projects/ --mode convos` until this is addressed; it will crash again on the same two files.