dotfiles/claude/memory/project-epub-repair-pipeline.md

---
name: EPUB → chamber-library conversion pipeline (with structural repair)
description: For scholarly EPUBs where pandoc alone loses chapter structure, repair
  the EPUB source first via TOC-driven heading promotion, then convert with pandoc
  + strip_cruft. Built 2026-05-13.
type: project
originSessionId: a5411fd1-4a96-43e7-aa1a-1a50d8951a8f
permalink: claude-memory/project-epub-repair-pipeline
---

For converting EPUBs from the messy folder (`~/Documents/___The Library [ePub_AWZ3]/`) into chamber-library, the durable pipeline is:

```
EPUB  →  repair_epub_headings.py  →  pandoc -t gfm-raw_html  →  strip_cruft.py  →  chamber-library
        (TOC-driven heading promotion)   (clean MD)             (final residue)
```

Each stage:

1. **`~/_Dev/chamber-library/scripts/repair_epub_headings.py`** — uses the EPUB's own NCX/NAV (machine-readable TOC) as authoritative source for chapter structure. Promotes styled `<p class="…">` chapter titles to proper `<h*>` tags. Publisher-convention-independent. Output is a new EPUB at a specified path; original is never modified.

   Usage: `repair_epub_headings.py INPUT.epub OUTPUT.epub [--max-depth N] [--verbose]`

   Uses dedicated venv at `~/.local/chamber-tools-venv/` (bs4 + lxml).

2. **`pandoc -t gfm-raw_html --wrap=preserve --extract-media=DIR -o OUT.md REPAIRED.epub`** — convert to clean Markdown. The `-raw_html` suffix suppresses HTML scaffolding preservation (the lesson from the Polastrons/Obrist work earlier same day).

3. **`~/_Dev/chamber-library/scripts/strip_cruft.py OUT.md --apply --samples 0`** — final cleanup pass for any pandoc class residue + link/image attribute stripping. Writes backup to `~/_Dev/chamber-library/_backups/YYYY-MM-DD/`.

**Tested 2026-05-13 on three different publisher conventions, all successful:**

- Rowman & Littlefield (Bachelard *Dialectic of Duration*): `<p class="CN">/<p class="CT">` encoding, anchor IDs in NCX. 19/20 promoted; final MD has 19 H1 headings including all 8 chapter titles ("Relaxation and Nothingness", etc.).
- Routledge (Weil *Gravity and Grace*): chapter-per-file encoding, single styled-title paragraph per file. 39/46 promoted (front matter skipped); final MD has 38 chapter H1 headings.
- Bloomsbury (Gadamer *Truth and Method*): `<p class="cn">/<p class="ct">` for chapters + `<p class="pn">/<p class="pt">` for parts, no fragments in NCX. 33/35 promoted; final MD has 10 H1 + 16 H2 = 26 headings (3 Parts + 5 main chapters + 6 Appendices + 2 Supplements + Translator's Preface).

**Why the tool is publisher-convention-independent:**
- Uses NCX/NAV (required by EPUB spec) as authoritative TOC source
- Matches by anchor id (preferred) OR by text content (fallback)
- Text matching handles leading numeration in many forms: arabic ("1 Title"), Roman ("III: Title"), English words ("Part One Title")
- Number-stub fall-through: when anchor target is just a chapter number (e.g., "Chapter 1"), falls through to text matching to find the substantive title
- Inner `<a>` wrapper stripping by default so pandoc produces clean `# Title` not `# [Title](anchor)`

**Important context — the failure mode this tool fixes:**

Many scholarly EPUBs (especially from typesetting workflows like Adobe InDesign export) encode chapter titles as styled paragraphs:
```html
<p class="CN">Chapter 1</p>
<p class="CT">Relaxation and Nothingness</p>
```
rather than:
```html
<h1>Chapter 1: Relaxation and Nothingness</h1>
```

When pandoc converts these with default flags, it sees only paragraphs — no headings to promote. Output is structurally flat regardless of the conversion flag. The chamber-library audit flagged 127 files in this state (RECONVERT bucket).

This tool fixes the source EPUB first so any subsequent conversion (now or later) inherits proper structure.

**Filing discipline (steward's directive 2026-05-13):**
- **EPUBs and MD conversions live in DIFFERENT locations** — never mix
- **Repaired EPUBs**: `~/Documents/__Chamber Source EPUBs/repaired-YYYY-MM-DD/<descriptive-name>.epub`
  - File apart from chamber-library entirely; this is source-material territory, not converted-text territory
  - Naming matches the chamber MD filename (e.g., `the_dialectic_of_duration_bachelard.epub` for chamber `the_dialectic_of_duration_bachelard.md`)
  - Double-underscore prefix on the parent folder follows steward's convention for important folders like `__must read/`
- **MD conversions**: filed at the canonical chamber-library path (e.g., `~/_Dev/chamber-library/converted_texts/traditions/critical_modernity/phenomenology/the_dialectic_of_duration_bachelard.md`)
- **Original EPUBs** in `~/Documents/___The Library [ePub_AWZ3]/` and `~/Desktop/New texts may 2026/` are never modified
- **Existing chamber file backups** (when replacing a broken existing MD): `~/_Dev/chamber-library/_backups/YYYY-MM-DD/<rel-path>.preconversion-backup`
- `mempalace mine` automatically detects mtime change on the chamber MD → deletes old drawers → re-mines clean content on its next iteration through that directory

**For the broader RECONVERT queue (127 files):**

This tool is the right shape for any EPUB-source RECONVERT file. For PDF-source files, docling is the conversion tool (per the toolset study 2026-05-13). For files where the source can't be found in `~/Documents/___The Library [ePub_AWZ3]/` or `~/Desktop/New texts may 2026/`, the file stays in the chamber as-is until source surfaces.