72 lines
12 KiB
Markdown
72 lines
12 KiB
Markdown
---
|
||
name: session-ledger-2026-06-27
|
||
description: "Practice-of-return ledger maintained by /symmetria — returns, open horizons, recalibrations, authorization moves, sub-agent dialogues, bypasses."
|
||
metadata:
|
||
node_type: memory
|
||
type: feedback
|
||
originSessionId: a17a5431-cfab-4beb-b9e6-5debcb06a5a4
|
||
---
|
||
|
||
# Session Ledger — 2026-06-27
|
||
|
||
## Returns
|
||
- 2026-06-27T11:50 — Held the inherited "Alexander needs the ToC+page-offset resolver" premise OPEN and measured book1 directly before designing. Premise overturned by the substrate: book1's chapters live IN THE BODY (`CHAPTER ONE`\n`THE PHENOMENON OF LIFE`…`CHAPTER ELEVEN`), under `PART ONE`/`PART TWO` — no offset resolver needed. The gnarly reprinted four-book ToC is front-matter NOISE (trim target), and the back-matter `Chapter 1…10` block is picture-credits apparatus (trim), NOT structure. (measure-before-theorizing; inherited-marker-read-as-current-state avoided.)
|
||
|
||
## Findings — literal-question answer (Alexander book1)
|
||
- Mechanical layer: normalize_ocr.py dry-run CLEAN (verbatim PASS, 0 deferred). Fits as-is.
|
||
- Structure layer: normalizer's running-head recovery is the WRONG tool here (13-item noise mix). The reliable structure is the body `CHAPTER N`+title anchors → needs the curatorial map, but a DIFFERENT shape than Levi's flat TITLE→offset: Alexander has 2-level PART→CHAPTER hierarchy + Preface/Conclusion/Appendix.
|
||
- Map unit: ONE PER VOLUME (book1: 11 ch / book2: 21 / book3: 19 / book4: 11 — each its own body+chapter set). The reprinted ToC is shared but it's a trim target, not a sharing opportunity.
|
||
|
||
## Open horizons
|
||
- 2026-06-27T11:39 — Pulling thread (inherited): apply proven OCR→canonical pipeline to Alexander Books 1–4. Literal Q held open: does Alexander's gnarlier structure (4-vol reprinted ToC, Roman-numeral chapters, per-book page restarts) fit the current per-book `structure.tsv` + normalizer, or need a richer instrument; one map per volume or shared across the four-book series. Answer empirically (book1 dry-run) before designing.
|
||
- `Bash(ssh capablehands:*)` NOT granted — Alexander pulls from CapableHands + driver note still hit a permission prompt; surface to steward before pulling books 1–3.
|
||
- REVIEWED-45 owed from steward (chamber spec ratification paper loop; PENDING-42 holds open voice-layer work).
|
||
- Engine re-ingest owed (coverage-ledger + chunk-quality stale to re-pointed Levi).
|
||
- Skill-harvest `/graduate-chamber-source` PROPOSED, awaiting authorization.
|
||
|
||
## Confidence to recalibrate
|
||
- 2026-06-27 — book1 graduation surfaced an OCR-quality issue NOT caught by the verbatim guard (PASS-BUT-FALSELY shape): ~8 of 491 pages hard-wrapped (olmOCR preserved physical line breaks → mid-word hyphens "real-/ize", "domi-/nate", a "be-/because" doubling) + ~87 total mid-word-hyphen breaks (clusters on the 8 pages + isolated singles on flowing pages). Words intact (guard PASS) but formatting broken → NOT graduation-clean. Localized, not pervasive (~98% clean). Also found: keepable front matter (dedication + the untitled authorial intro = ToC's "The Art of Building…", printed p.1). Fork put to steward: (A) post-process paragraph-reflow in normalize [risk: entangles with heading-block detection — "CHAPTER ONE\\nTITLE" is itself a consecutive-non-blank block]; (B) re-OCR the 8 pages via driver --pages splice [tool supports it; may re-hard-wrap]; (C) fold into the planned post-batch reconversion (book1 IS a Making source). Lean B/C over A.
|
||
|
||
## Authorization moves
|
||
- 2026-06-27 — Steward authorized fork "B": teach the pipeline the 2-level PART→CHAPTER shape (durable, recurs across all 4 Alexander vols). Two governed-tool edits made [HARDENING-class, steward-directed]:
|
||
1. insert_chapter_headings.py — optional LEVEL column (default 2) + unified REPLACE-BLOCK (replaces a printed-heading block: Levi's bare `CONCLUSION` line AND Alexander's `CHAPTER N`+title block). Levi regression: BYTE-IDENTICAL to old tool.
|
||
2. normalize_ocr.py — `--no-heading-recovery` (keeps runhead DEDUP, skips PROMOTION) so curatorial-structure works get clean prose + no spurious/wrong-level auto-headings. Levi default-path regression: BYTE-IDENTICAL. book1 with flag: 0 auto-headings, verbatim PASS (164511 tokens), 11 CHAPTER markers survive.
|
||
3. normalize_ocr.py REFINED (steward fork "B", prime-directive build-once): `detect_running_heads` now counts PAGE-TOP occurrences only (first non-blank after `## Page N`) — true furniture recurs page-top; mid-content section labels (NOTES/FUNCTIONAL NOTES/PART) open mid-page → preserved. Levi default BYTE-IDENTICAL. book1: 0 spurious headings, all 26 note-labels + PART dividers + 11 CHAPTER markers survive, guard PASS (164,683 tok).
|
||
- FINDING (books 2–4 + OCR driver): olmOCR clean-prose prompt already strips per-page furniture → page-top finds 0 in book1 → default == --no-heading-recovery; flag kept as explicit safety. Alexander structure = front/back trim + curatorial structure.tsv only.
|
||
- Tool-evolution-log entry WRITTEN (feedback-tool-review-after-each-use): `_curation/tool-evolution-log.md` 2026-06-27.
|
||
- OPEN finishing choice (steward): note-labels render `### Notes`/`### Functional Notes` (faithful, needs all-match promotion — non-unique anchors) vs plain caps verbatim (citation-neutral, no new tooling). Affects all 4 vols.
|
||
|
||
## DECISION FORK surfaced (part-level anchoring) — awaiting steward
|
||
- Suppression drops the bare `PART ONE/TWO` body dividers (they recur 4× via the reprinted 4-book ToC → classed as running heads). PREFACE/CONCLUSION/APPENDICES + the 11 CHAPTER markers all survive. So part-level anchors need a decision: (a) trim front-matter BEFORE normalize (then PART recurs only 1–2× → survives) — changes pipeline order; or (b) anchor "Part One" via insert before the Chapter-One block; or (c) accept parts as un-headed and carry only chapters (parts live in the ToC/citation only).
|
||
|
||
## Sub-agent dialogues
|
||
|
||
## B-test result (hard-wrap fix)
|
||
- 2026-06-27 — Re-OCR'd page 437 (worst hard-wrap, 15 breaks) via /tmp/olmocr_reflow.py (driver + explicit reflow/de-hyphenate prompt) on CapableHands. Result: PARTIAL + unreliable — joined some wrapped lines but left others, STILL produced mid-word hyphens. olmOCR-7B is layout-faithful, ignores reflow meta-instructions. **B fails; C (same engine) would too.** Only reliable fix = post-process reflow (A) or a different OCR engine for the 8 pages. Test driver left at capablehands:/tmp/olmocr_reflow.py.
|
||
|
||
## Within-page rejoin built (A) + a corpus-integrity discovery
|
||
- 2026-06-27 — normalize_ocr.py: extracted seam-join into join_wrapped()/record_join() helpers (seam path unchanged), added WITHIN-PARAGRAPH rejoin (lowercase-continuation rule; uppercase/blank/runhead/pagenum stop it → headings/dedication/verse safe). book1: 464 in-page joins, 2583 region now flowing prose, verbatim PASS, 11 CHAPTER markers intact; residual = 14 boundary/seam hyphens + 4 DEFERRED compounds (whole-ess/bounded-edness/left-hand/geo-métrical → human review per design).
|
||
- **DISCOVERY (load-bearing):** the rejoin fired 30× on Levi → Levi's PUBLISHED canonical carries latent hard-wrap defects (hard-wrapped prose + mid-word hyphens "every-/one", "pro-/found", "real-/ized"). All 30 changes validated as legitimate reflows. The "excellent precedent" shipped with this; words intact so the guard passed. → byte-identity is NO LONGER the validation gate (the fix legitimately changes output); gate is now verbatim-guard + reflow-review.
|
||
- Corpus scan (hyphen-ending signature): concentrates in `_loeb_bilingual/dsl_clean/*` (Plautus/Terence/Euripides — DIFFERENT pipeline, likely legitimate VERSE line-breaks, NOT this defect → do NOT reflow). The olmOCR-hard-wrap defect is narrower (Levi + the small olmOCR-graduated set); grep undercounts it (catches only hyphen-ends, not non-hyphen wraps).
|
||
- Implication: re-graduate Levi + audit the olmOCR-graduated set = bounded follow-on (steward-authorized, separate from book1). Loeb files OUT of scope (verse).
|
||
|
||
## RETURN — recalibration (premature alarm)
|
||
- 2026-06-27 — Drifted to "content corruption / book1 needs reconversion" from ONE page (L1809: footnote-interleave + re-transcribed paragraph-opening) BEFORE measuring scope. Measurement corrected it: 0 gross duplication; damage = the SAME 8 of 491 pages (98% clean). Localized + hand-repairable, not whole-book. Contamination shape: causal-story/alarm before census. Recalibrated to steward honestly + owned it. book1 status: 483 pages clean (reflowed); 8 figure/footnote-dense pages need curatorial repair (split-word rejoin across captions/footnotes, drop duplicated restart, footnote placement). 3 OCR char-errors already corrected+documented (wholeness/boundedness/geometrical). Fork to steward: (a) hand-repair 8 pages [recommended], (b) ocrmac re-OCR those 8. Graduation files staged: converted_texts/.../nature-of-order-vol-1...md (raw §III) + /tmp/book1.clean.md (reflowed+corrected body) + conversion.yaml.
|
||
|
||
## Landing state — ocrmac column-aware conversion toolset (steward: option 2, "this has to be perfect")
|
||
- 2026-06-27 — Pivoted from patching to BUILDING a robust ocrmac conversion toolset (steward mandate: take the time, find the method, take notes, cover all OCR challenges; this is the engine's foundational corpus).
|
||
- METHOD DOC: `chamber-library/docs/ocr-conversion-method-2026-06-27.md` (the notes — problem/engine/detection/pipeline/open-challenges/validation).
|
||
- ROOT CAUSE: book1 is PREDOMINANTLY 2-column; olmOCR reflows per-page → reads clean 2-col correctly (masking the layout) but SCRAMBLES figure-complicated 2-col (reads across the gutter). Word-guard can't catch (same words, wrong order).
|
||
- ENGINE: ocrmac (Apple Vision) — per-line text+bbox+confidence, we control reassembly. Installed on CapableHands (ensurepip→pip install ocrmac; render via fitz). Column-major reassembly PROVEN correct + more accurate than olmOCR on p437/107/247.
|
||
- COLUMN DETECTION: vertical-projection profile is the method (OCR-geometry unreliable — ocrmac over-segments → false signals; caused a wrong "p42=1col" call the STEWARD corrected by reading the page image. Lesson: ground-truth layout vs the image, not OCR boxes). Detector finds true-2col gutters (p42/437/225) BUT naive threshold MISSES bridged-gutter (p226) + figure-occupied-column (p247) pages — the calibration gap, planned in §6b.
|
||
- STATUS: foundation laid, detector prototyped + limits mapped. NOT yet built: calibrated detector → driver (ocrmac_columns.py) → full book1 re-OCR → verify → graduate → books 2-4. Multi-session build.
|
||
- OWED: Levi re-graduation (latent hard-wrap defect); book1 graduation staged but SUPERSEDED by the re-OCR decision (don't graduate the olmOCR version). PENDING-42/REVIEWED-45 (chamber spec) still open. Page-number preservation to fold INTO the new driver (steward's banked enhancement).
|
||
|
||
## CALIBRATION SETTLED (evening) — both halves
|
||
- COLUMN DETECTION solved: 150dpi+accurate (DPI was the key — 120 merged lines across gutter), gutter via line-crossing of narrow lines + line-width classifier (medW≈0.38/narrowfrac≥0.92 = 2col; figure-dominant pages p179/226/247 now handled). All 491 pages classified (386 2col/30 1col/75 sparse). Reassembly coherent (p226 Yanagi reads in order; aspact OCR error gone). Full OCR cached → capablehands:/tmp/book1_ocr.json (+ pulled /tmp/book1_ocr.json).
|
||
- DE-HYPHENATION solved + MULTILINGUAL: normalize_ocr.py now uses wordfreq (pip install --break-system-packages wordfreq) — merged-Zipf≥3.4→merge / both-parts≥4.0→compound / unknown-lowercase→merge+FLAG. `--lang en|fr|de|es` (steward's forward req — Camus/Handke covered). Separates split-common-word from real-compound (impossible by membership). book1: 3008 merges/131 compounds/109 flagged(3.6%, real OCR+reflow errors). Levi verbatim PASS.
|
||
- normalize_ocr.py is now a SIGNIFICANT governed-tool evolution (join_wrapped/record_join helpers + within-page rejoin + page-top runhead detection + --no-heading-recovery + wordfreq multilingual de-hyph + merge_flag). All regression-checked.
|
||
- NEXT: package the ocrmac pipeline as a governed driver (render→ocr→detect→reassemble→furniture: running-heads/page-numbers[preserve per steward]/captions/footnotes), run book1 end-to-end, graduate, books 2-4. Owed: Levi re-graduation.
|
||
|
||
## Bypasses
|