dotfiles/claude/memory/project-mempalace-chunking-bug-2026-05-16.md

---
name: mempalace-chunk-by-paragraph-oversized-chunk-bug-diagnostic-record-2026-05-16
description: 'Specific mempalace bug identified the night of 2026-05-16 after a mine
  attempt of ~/.claude/projects/ failed with 120 GiB attention buffer error. Root
  cause: _chunk_by_paragraph (convo_miner.py:184-186 + 176-182) does not enforce CHUNK_SIZE,
  letting paragraphs become single oversized chunks that crash the embedding model.
  Reproducer + line numbers + fix shape recorded; decision pending on whether to file
  upstream as GH issue.'
metadata:
  node_type: memory
  type: project
  originSessionId: 8952fe77-1cf0-401d-9d88-01f9f0b13bcc
permalink: claude-memory/project-mempalace-chunking-bug-2026-05-16
---

# MemPalace `_chunk_by_paragraph` oversized-chunk bug

**Discovered:** 2026-05-16 night
**Status:** FILED upstream as MemPalace/mempalace#1534 (2026-05-17) — https://github.com/MemPalace/mempalace/issues/1534
**Follow-up evidence posted:** 2026-05-17 — https://github.com/MemPalace/mempalace/issues/1534#issuecomment-4469591465 (broader scope: both bug locations firing in practice; tool-results .txt files with 834 KB single lines hit the line-group fallback; convos-mode mining is effectively blocked on any real Claude Code corpus). Severity framed toward "high" in the follow-up.
**Resolved-locally:** No (per steward direction: "not fix someone else's work"); waiting on upstream fix. Operations 2/3/4 of the four-op plan remain held until upstream ships.
**Reproduction note (2026-05-17 morning):** Quarantining the two original-repro files (`e31838f5` + `8952fe77` → `.jsonl.QUARANTINE-2026-05-17`) and re-running the mine crashed again on the next non-skipped file in iteration order with identical 120 GiB upsert error. Restored files cleanly; palace state unchanged at 14,496 drawers. **Side effect:** during the brief quarantine window (~30 seconds), Claude Code recreated `8952fe77-...jsonl` with in-flight session content; the restore-via-mv overwrote that recreation. The few minutes of session-turn writes in that window are missing from the on-disk transcript (session memory file + this bug record capture the meaningful conclusions). Recoverable but worth knowing if future wake-up reads of 8952fe77.jsonl show a content gap.

---

## Summary

`mempalace mine <dir> --mode convos` can crash with `RuntimeError: Invalid buffer size: 120.00 GiB in upsert.` when a session jsonl file contains content that gets routed through `_chunk_by_paragraph` (the fallback chunker) AND has a paragraph longer than what the embedding model's attention computation can hold. Today's reproducer is a Claude Code session jsonl containing a large pasted document (~30KB) as a single user turn — the paragraph chunker doesn't split it; embedding tries to compute attention on a single oversized chunk and runs out of memory.

The bug is localized to **two specific code paths** in `convo_miner.py`. The peer chunker `_chunk_by_exchange` handles size correctly. Storage layer, embedding layer, MCP, hooks, search, KG — all working. The damage is bounded to this one path.

---

## Repro

**Environment:**
- mempalace local HEAD: `604df3b local: raise MAX_CHUNKS_PER_FILE 500→50_000 for full-text scholarly corpora` (note: MAX_CHUNKS_PER_FILE is in `miner.py`, not relevant to this bug; mentioned only for environment fingerprint)
- Python 3.13, embedding model: xlm-roberta variant (391 weights loaded per startup banner)
- macOS Apple Silicon, palace at `~/.mempalace/palace-memory`

**Trigger:**
```
mempalace mine ~/.claude/projects/ --mode convos --wing claude-sessions --agent claude-code
```

**Observed:**
- First file processed: traceback as below
- Zero drawers added on the failed run
- Palace state unchanged at 14,494 drawers (already populated by an earlier same-day successful mine that ran 16:47–17:47 before the trigger files appeared)

**Traceback (top of stack):**
```
File "/Users/davidglidden/_Dev/mempalace/mempalace/convo_miner.py", line 489, in mine_convos
    drawers_added, room_delta, skipped = _file_chunks_locked(
File "/Users/davidglidden/_Dev/mempalace/mempalace/convo_miner.py", line 370, in _file_chunks_locked
    collection.upsert(
        documents=batch_docs,
…
File ".../transformers/integrations/sdpa_attention.py", line 92, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(...)
RuntimeError: Invalid buffer size: 120.00 GiB in upsert.
```

Full failing-run log preserved at `/private/tmp/claude-501/-Users-davidglidden/8952fe77-1cf0-401d-9d88-01f9f0b13bcc/tasks/bm45temj3.output` (172 lines, the relevant lines are start banner + the upsert traceback).

**The specific input that triggers it (2026-05-16):**

Two Claude Code session jsonl files created today:
- `~/.claude/projects/-Users-davidglidden/e31838f5-238e-446f-888d-7adab6eaf318.jsonl` — 2.88 MB, 460 lines, max line length 135,600 chars
- `~/.claude/projects/-Users-davidglidden/8952fe77-1cf0-401d-9d88-01f9f0b13bcc.jsonl` — 2.10 MB, 618 lines, max line length 119,671 chars

These contain large pasted content (an ADR document paste was ~30KB; Read tool results returned ~100KB markdown files multiple times) as single jsonl lines.

**Dry-run chunk counts as a smoking-gun comparison:**

| File | Source | Chunks | Avg chunk size |
|---|---|---|---|
| `a156edbb-...jsonl` | older Claude Code session, similar wall-clock size | 1352 | ~2 KB |
| `e31838f5-...jsonl` | TODAY (20:31) | **170** | **~17 KB** |
| `8952fe77-...jsonl` | TODAY (22:18) | **135** | **~15 KB** |

The older file went through `_chunk_by_exchange` and produced safe 2KB chunks. The two new files went through `_chunk_by_paragraph` and produced massively oversized chunks. The `_chunk_by_exchange` vs `_chunk_by_paragraph` dispatch is decided by `quote_lines >= 3` check at `convo_miner.py:108` — whether the normalized content has ≥3 lines starting with `>`. Why the two new files don't meet that threshold needs verification but is likely a `normalize()` behavior difference for very-recent Claude Code jsonl format.

---

## Bug location

`mempalace/convo_miner.py`:

### Bug 1 — paragraph chunking (lines 184–186)

```python
def _chunk_by_paragraph(content: str) -> list:
    """Fallback: chunk by paragraph breaks."""
    chunks = []
    paragraphs = [p.strip() for p in content.split("\n\n") if p.strip()]

    # If no paragraph breaks and long content, chunk by line groups
    if len(paragraphs) <= 1 and content.count("\n") > 20:
        lines = content.split("\n")
        for i in range(0, len(lines), 25):
            group = "\n".join(lines[i : i + 25]).strip()
            if len(group) > MIN_CHUNK_SIZE:
                chunks.append({"content": group, "chunk_index": len(chunks)})
        return chunks

    for para in paragraphs:
        if len(para) > MIN_CHUNK_SIZE:
            chunks.append({"content": para, "chunk_index": len(chunks)})  # ← BUG: no CHUNK_SIZE cap

    return chunks
```

A paragraph above `CHUNK_SIZE` (800 chars) is appended as a single oversized chunk. No splitting.

### Bug 2 — line-group fallback (lines 178–181)

Same shape: groups 25 lines at a time, doesn't enforce `CHUNK_SIZE` on the resulting group.

### Contrast — the working path (lines 145-156, `_chunk_by_exchange`)

```python
if len(content) > CHUNK_SIZE:
    first_part = content[:CHUNK_SIZE]
    if len(first_part.strip()) > MIN_CHUNK_SIZE:
        chunks.append({"content": first_part, "chunk_index": len(chunks)})
    remainder = content[CHUNK_SIZE:]
    while remainder:
        part = remainder[:CHUNK_SIZE]
        remainder = remainder[CHUNK_SIZE:]
        if len(part.strip()) > MIN_CHUNK_SIZE:
            chunks.append({"content": part, "chunk_index": len(chunks)})
```

This is the correct shape. The fix for `_chunk_by_paragraph` is to mirror this splitting loop in both bug locations.

---

## Suggested fix (sketch, not committed)

In `_chunk_by_paragraph`, both paths need a size-bounded splitter. Suggested helper:

```python
def _emit_bounded(chunks: list, content: str) -> None:
    """Append content as one or more chunks, none exceeding CHUNK_SIZE."""
    while content:
        part = content[:CHUNK_SIZE]
        content = content[CHUNK_SIZE:]
        if len(part.strip()) > MIN_CHUNK_SIZE:
            chunks.append({"content": part, "chunk_index": len(chunks)})
```

Then replace both bug sites:

```python
# Line ~180 (line-group fallback):
for i in range(0, len(lines), 25):
    group = "\n".join(lines[i : i + 25]).strip()
    _emit_bounded(chunks, group)

# Line ~184 (paragraph loop):
for para in paragraphs:
    _emit_bounded(chunks, para)
```

This makes `_chunk_by_paragraph` symmetric in size-discipline with `_chunk_by_exchange`.

---

## Honest accounting of what is known vs hypothesized

### Known with high confidence

- `_chunk_by_paragraph` does not enforce `CHUNK_SIZE`. Code is verbatim above; the absence of splitting is visible.
- The two new today-evening jsonl files produced anomalous chunk counts (170 and 135) on dry-run; older similar-size files produced 1000+ chunks. The ratio is the smoking gun.
- The mine failed with a 120 GiB attention buffer error on the first non-skipped file's upsert.
- `_chunk_by_exchange` handles size correctly (lines 145-156).
- Earlier same-day mine (16:47–17:47) succeeded across ~245 files; only the two new files added after 17:47 are candidates for triggering the crash.

### Hypothesis (high confidence but not direct-verified)

- The exact failing file is one of `e31838f5-...` or `8952fe77-...`. Dry-run chunk-counts strongly imply both would crash on embedding, but the live mine failed on the FIRST non-skipped file; whichever the iterator hit first is the one that crashed. Not directly checked.
- The 120 GiB number comes from attention being O(seq_len²) and the embedding model trying to handle a 30K–50K-token-equivalent single chunk. The exact arithmetic depends on model + batch + tokenization specifics; the order-of-magnitude fits a 100KB-paragraph chunk.

### Open questions worth checking before filing

1. **Why don't the two new jsonl files trigger the `quote_lines >= 3` branch?** Are recent Claude Code jsonl files normalized differently? Could be a recent `normalize()` change, or just that today's sessions happen to lack `>`-prefixed quotes. Worth checking by inspecting `normalize()` output on the two files (the verification Python script tonight failed on a missing chromadb module in the system Python; needs to run inside mempalace's venv).
2. **Was the chunking bug present in earlier mempalace versions, or recent?** Worth a git-blame on `_chunk_by_paragraph`. If recent, more urgent.
3. **Does `--mode projects` (miner.py) have the same shape of bug in its chunker?** Operations 2/3/4 of the four-op plan would use projects mode; needs separate verification.
4. **Why did the earlier-today 16:47 mine succeed?** Probably because (a) it ran before the two new jsonl files appeared, and (b) older jsonl files all went through `_chunk_by_exchange`. Both verifiable from dry-run output.

### What this is NOT

- NOT a storage corruption bug. The 14,494 drawers already in `palace-memory` are intact and searchable.
- NOT a re-mining-duplicates bug. `file_already_mined` skips already-ingested files before any upsert; duplicate "already exists" errors are caught at upsert. (My first hypothesis tonight conflated these; corrected after steward push-back.)
- NOT a `MAX_CHUNKS_PER_FILE` issue. That constant lives in `miner.py` (project files), not `convo_miner.py`.
- NOT a chromadb / embedding-model installation bug. Embedding works for normal-sized chunks; only oversized ones crash.

---

## If filing as a GitHub issue

**Title suggestion:** `convo_miner: _chunk_by_paragraph does not enforce CHUNK_SIZE, can produce single oversized chunks that crash embedding`

**Body skeleton:**
- Summary (one paragraph, the bug shape)
- Reproducer (env + the dry-run chunk-count comparison)
- Code locations (convo_miner.py:184-186, 176-182)
- Suggested fix (the `_emit_bounded` helper, both call-sites)
- What's affected: `mempalace mine --mode convos` against any directory containing jsonl files with large single-line content that doesn't trigger the `quote_lines >= 3` branch

**Severity framing:** medium-high — the failure is loud (exception) rather than silent-corruption, but it stops a long-running operation cold and the user wastes the embedding setup time. With trust-cost factored in (a steward who has lost weeks to mining time), severity reads higher.

**Repo to file against:** MemPalace/mempalace — same repo where #1526 (cascade + silent-empty-index) was filed earlier today.

---

## Trust-and-decision context

The steward stated "I need to be able to trust mempalace" tonight after losing weeks to mining-related issues. This bug is specific, bounded, reproducible, and mechanically fixable. The damage from this particular bug is bounded to:
- The chunker fallback path (one function, two lines each)
- Sessions where normalized content lacks `>`-quote markers AND has paragraphs above CHUNK_SIZE

The broader trust concern is mining-time-vs-failure-mode visibility: long mines that fail near the end (or, here, on the first non-skipped file after a setup) are expensive even when they fail loudly, and exit-0-via-`tee` masking of the underlying nonzero exit (a shell pattern, my error tonight, not mempalace's) compounds this.

Decision recommendation for steward (offered, not made):
- **File the GH issue** if you want this fixed upstream and shipped to other users. Bug is concrete; fix is small; report would be useful.
- **Defer filing + fix locally** if you want immediate working ingestion for ops 2/3/4 without waiting on upstream merge. The `_emit_bounded` helper is ~5 minutes of work in a fresh head.
- **Defer + accept current corpus as-is** if you want to use what's already in palace-memory and not push more ingestion until upstream ships a fix.

Either way: don't retry `mempalace mine ~/.claude/projects/ --mode convos` until this is addressed; it will crash again on the same two files.