dotfiles/claude/memory/project-epub-repair-pipeline.md

name, description, type, originSessionId

name	description	type	originSessionId
EPUB → chamber-library conversion pipeline (with structural repair)	For scholarly EPUBs where pandoc alone loses chapter structure, repair the EPUB source first via TOC-driven heading promotion, then convert with pandoc + strip_cruft. Built 2026-05-13.	project	a5411fd1-4a96-43e7-aa1a-1a50d8951a8f

For converting EPUBs from the messy folder (~/Documents/___The Library [ePub_AWZ3]/) into chamber-library, the durable pipeline is:

EPUB  →  repair_epub_headings.py  →  pandoc -t gfm-raw_html  →  strip_cruft.py  →  chamber-library
        (TOC-driven heading promotion)   (clean MD)             (final residue)

Each stage:

1. ~/_Dev/chamber-library/scripts/repair_epub_headings.py — uses the EPUB's own NCX/NAV (machine-readable TOC) as authoritative source for chapter structure. Promotes styled <p class="…"> chapter titles to proper <h*> tags. Publisher-convention-independent. Output is a new EPUB at a specified path; original is never modified.

   Usage: repair_epub_headings.py INPUT.epub OUTPUT.epub [--max-depth N] [--verbose]

   Uses dedicated venv at ~/.local/chamber-tools-venv/ (bs4 + lxml).

2. pandoc -t gfm-raw_html --wrap=preserve --extract-media=DIR -o OUT.md REPAIRED.epub — convert to clean Markdown. The -raw_html suffix suppresses HTML scaffolding preservation (the lesson from the Polastrons/Obrist work earlier same day).

3. ~/_Dev/chamber-library/scripts/strip_cruft.py OUT.md --apply --samples 0 — final cleanup pass for any pandoc class residue + link/image attribute stripping. Writes backup to ~/_Dev/chamber-library/_backups/YYYY-MM-DD/.

Tested 2026-05-13 on three different publisher conventions, all successful:

• Rowman & Littlefield (Bachelard Dialectic of Duration): <p class="CN">/<p class="CT"> encoding, anchor IDs in NCX. 19/20 promoted; final MD has 19 H1 headings including all 8 chapter titles ("Relaxation and Nothingness", etc.).
• Routledge (Weil Gravity and Grace): chapter-per-file encoding, single styled-title paragraph per file. 39/46 promoted (front matter skipped); final MD has 38 chapter H1 headings.
• Bloomsbury (Gadamer Truth and Method): <p class="cn">/<p class="ct"> for chapters + <p class="pn">/<p class="pt"> for parts, no fragments in NCX. 33/35 promoted; final MD has 10 H1 + 16 H2 = 26 headings (3 Parts + 5 main chapters + 6 Appendices + 2 Supplements + Translator's Preface).

Why the tool is publisher-convention-independent:

• Uses NCX/NAV (required by EPUB spec) as authoritative TOC source
• Matches by anchor id (preferred) OR by text content (fallback)
• Text matching handles leading numeration in many forms: arabic ("1 Title"), Roman ("III: Title"), English words ("Part One Title")
• Number-stub fall-through: when anchor target is just a chapter number (e.g., "Chapter 1"), falls through to text matching to find the substantive title
• Inner <a> wrapper stripping by default so pandoc produces clean # Title not # [Title](anchor)

Important context — the failure mode this tool fixes:

Many scholarly EPUBs (especially from typesetting workflows like Adobe InDesign export) encode chapter titles as styled paragraphs:

<p class="CN">Chapter 1</p>
<p class="CT">Relaxation and Nothingness</p>

rather than:

<h1>Chapter 1: Relaxation and Nothingness</h1>

When pandoc converts these with default flags, it sees only paragraphs — no headings to promote. Output is structurally flat regardless of the conversion flag. The chamber-library audit flagged 127 files in this state (RECONVERT bucket).

This tool fixes the source EPUB first so any subsequent conversion (now or later) inherits proper structure.

Filing discipline (steward's directive 2026-05-13):

• EPUBs and MD conversions live in DIFFERENT locations — never mix
• Repaired EPUBs: ~/Documents/__Chamber Source EPUBs/repaired-YYYY-MM-DD/<descriptive-name>.epub
  • File apart from chamber-library entirely; this is source-material territory, not converted-text territory
  • Naming matches the chamber MD filename (e.g., the_dialectic_of_duration_bachelard.epub for chamber the_dialectic_of_duration_bachelard.md)
  • Double-underscore prefix on the parent folder follows steward's convention for important folders like __must read/
• MD conversions: filed at the canonical chamber-library path (e.g., ~/_Dev/chamber-library/converted_texts/traditions/critical_modernity/phenomenology/the_dialectic_of_duration_bachelard.md)
• Original EPUBs in ~/Documents/___The Library [ePub_AWZ3]/ and ~/Desktop/New texts may 2026/ are never modified
• Existing chamber file backups (when replacing a broken existing MD): ~/_Dev/chamber-library/_backups/YYYY-MM-DD/<rel-path>.preconversion-backup
• mempalace mine automatically detects mtime change on the chamber MD → deletes old drawers → re-mines clean content on its next iteration through that directory

For the broader RECONVERT queue (127 files):

This tool is the right shape for any EPUB-source RECONVERT file. For PDF-source files, docling is the conversion tool (per the toolset study 2026-05-13). For files where the source can't be found in ~/Documents/___The Library [ePub_AWZ3]/ or ~/Desktop/New texts may 2026/, the file stays in the chamber as-is until source surfaces.