Files
dotfiles/scripts/wake-digest.py
T
David F Glidden 32b0c9dfc6 [FIX] The link canary was blind to code spans, to wikilinks, and to its own class
The steward's 2026-08-09 to-do read "the gap is neither knowledge nor home but
the absence of an EXECUTABLE." The premise was false: classify_pointers has
existed since 19bddd5 (2026-08-08), wired to SessionStart, with controls. The
gap was that the executable was incomplete, and the incompleteness had already
produced a false positive.

Four defects, three named in the spec and one found by building it:

1. CODE SPANS. `](file.md)` inside backticks read as a pointer, so the single
   DEAD pointer reported on 2026-08-09 was the link pattern written inside
   MEMORY.md's own specification of this canary. An instrument that flags its
   own documentation flags it every wake forever, and the real signal drowns —
   the same "known canary bug" dismissal the 2026-07-28 block was written to
   end, arriving by a second route. Fences and inline spans are blanked with
   offsets preserved; inline spans may not cross a newline and an unterminated
   fence does not match, so a stray backtick can never blank the file and HIDE
   dead pointers.

2. WIKILINKS. reference-verification-ladder.md has specified this canary as
   covering "every `](file.md)` and `[[wikilink]]`" since 2026-07-06. Only the
   first half was ever built. 31 wikilinks now checked.

3. BREAKAGE AGE, derived from git rather than a stored prior run — a state file
   would make this the one cached section in a digest whose governing property
   is that it is computed. Where git cannot answer, it says so.

4. Found by running it: the first wikilink pass reported only UNWRITTEN, and
   both live hits were [[trust-prior-pass-frame]], whose file EXISTS as
   feedback-trust-prior-pass-frame.md. That is precisely the one-word alarm the
   comment ten lines above it was written to forbid. Wikilinks now report three
   outcomes and hand back the replacement slug. Both are repaired here.

The wake-up skill and the ladder now POINT AT the executable instead of
describing the check — the described-not-invoked gap is why it kept being
retyped by hand on 2026-08-08 and 2026-08-09.

Verify: python3 scripts/wake-digest.py --selftest   (61 checks, exit 0)
        python3 scripts/wake-digest.py | grep 'MEMORY POINTERS'
Induced red: blank_code reverted to a no-op (behaviour, not the symbol) →
        exit 2, five named failures, no traceback; direction controls held.

Not changed: the wrap_inside detector, which announced "PREVIOUS SESSION DID
NOT WRAP" for a session that wrapped at 19:48 and kept working until 21:54 —
a two-valued detector over a three-case state. Named in the ledger, not fixed.
2026-08-10 16:01:51 +02:00

934 lines
42 KiB
Python
Executable File

#!/usr/bin/env python3
"""Wake digest — cheap, computed session-start context.
Emits ~600 tokens of CURRENT state at SessionStart, replacing the ~1,000-token
session-handoff.md snapshot that had gone five months stale.
The governing property: this is DERIVED, never CACHED. A stored digest drifts
silently from the substrate — that is precisely how MemPalace's boot context
failed. A computation over the substrate cannot drift; it can only have bugs,
and bugs are found by positive controls. Every section here degrades honestly:
if it cannot be computed it says so rather than emitting nothing.
wake-digest.py emit the digest
wake-digest.py --selftest prove each extractor detects presence AND absence
Provenance: 2026-07-28, steward-authorized alongside the PENDING split and the
CLAUDE.md doctrine annotation. Sibling of governance-drift-check.py.
"""
import os, re, subprocess, sys, time
HOME = os.path.expanduser("~")
D = os.path.join(HOME, "dotfiles")
MEM = os.path.join(HOME, ".claude", "projects", "-Users-davidglidden", "memory")
PENDING = os.path.join(D, "PENDING.md")
REVIEWED = os.path.join(D, "REVIEWED.md")
DRIFT = os.path.join(D, "scripts", "governance-drift-check.py")
REPOS = ["CapableMind-AI", "BetterMemories.io", "chamber-library",
"animal-davidglidden-eu", "studium-engine"]
warn = []
def sh(args, cwd=None, timeout=8):
try:
r = subprocess.run(args, cwd=cwd, capture_output=True, text=True, timeout=timeout)
return r.stdout.strip() if r.returncode == 0 else None
except Exception:
return None
def read(p):
try:
return open(p, encoding="utf-8").read()
except Exception:
return None
# ---------- extractors (pure functions of text, so --selftest can exercise them) ----------
def item_spans(text):
"""-> [(header, start, end)] for every item; 1-based start, end exclusive.
THE single definition of "an item" for every consumer. Two lessons are welded in:
- **Any `## ` header is an item.** A family-specific regex (`^## PENDING-<digits>`)
hid twenty items on 2026-07-28, ten of them open. Never match on family.
- **Fenced blocks are not the document's structure.** Governance drafts are
written as plain fenced markdown carrying their own `## REVIEWED-N` headers
(the steward's copy-paste-clean practice), so a fenced header would otherwise
register as a phantom item AND truncate the real item containing it. Zero such
headers exist in the substrate today; the trigger is a drafting habit already
in use, so this is a defect waiting on the next draft rather than a hypothetical.
"""
lines = text.split("\n")
heads, fence = [], False
for i, l in enumerate(lines):
s = l.lstrip()
if s.startswith("```") or s.startswith("~~~"):
fence = not fence
continue
if not fence and l.startswith("## "):
heads.append((l[3:].strip(), i + 1))
return [(h, s, heads[k + 1][1] if k + 1 < len(heads) else len(lines) + 1)
for k, (h, s) in enumerate(heads)]
def open_items(text):
"""-> [(header, line_no)] for every item lacking closure evidence. Closure
evidence = a REVIEWED-N (resolved by the caller) or CLOSED/COMPLETED in the
header."""
return [(h, s) for h, s, _e in item_spans(text)
if "CLOSED" not in h and not h.startswith("COMPLETED")]
def tag_of(text, header):
m = re.search(re.escape("## " + header) + r".*?\n\*\*Tag:\*\*\s*(\[[A-Z]+\])",
text, re.S)
return m.group(1) if m else ""
def strip_frontmatter(text):
"""YAML frontmatter restates the thread inside `description:`, and matching
there runs the extraction on into `metadata:`. The body is the record."""
m = re.match(r"^---\n.*?\n---\n", text, re.S)
return text[m.end():] if m else text
def extract_anchor(text, anchor, limit=900):
"""Verbatim text following an anchor. Never summarised — a digest that
paraphrases the steward's own words is the lossy extraction this whole
architecture is designed to avoid.
Handles both wrap shapes: `**ANCHOR: content**` (label and content share the
bold run) and `**ANCHOR for next-Claude:** content` (label closes first)."""
text = strip_frontmatter(text)
i = text.find(anchor)
if i < 0:
return None
s = text[i + len(anchor): i + len(anchor) + limit + 120]
c = s.find(":")
if 0 <= c <= 40: # remainder of the label, not content
s = s[c + 1:]
cut = re.search(r"\n\s*\n", s)
if cut:
s = s[: cut.start()]
s = re.sub(r"\s+", " ", s.replace("**", "")).strip(" *—-")
return (s[:limit].rstrip() + " …[truncated]") if len(s) > limit else s
# ---------- sections ----------
def sec_pause():
files = [f for f in os.listdir(MEM)
if f.startswith("session-") and not f.startswith("session-ledger-")]
if not files:
warn.append("no session memory files found")
return None, None
newest = max(files, key=lambda f: os.path.getmtime(os.path.join(MEM, f)))
age = time.time() - os.path.getmtime(os.path.join(MEM, newest))
h = age / 3600
span = f"{age/60:.0f} min" if h < 1 else (f"{h:.1f} h" if h < 48 else f"{h/24:.1f} days")
return newest, span
def wrap_records():
"""Wrap records only — session-ledger-*.md is a Symmetria artifact, not a wrap."""
return [os.path.join(MEM, f) for f in os.listdir(MEM)
if f.startswith("session-") and not f.startswith("session-ledger-")]
def transcript_span(path):
"""
A session's true span, from the timestamps INSIDE the transcript — never mtime.
mtime says when the file was last touched; it cannot say when the session ran.
Returns (first_epoch, last_epoch), or (None, None) if unreadable.
"""
first = last = None
try:
with open(path, errors="ignore") as f:
for line in f:
m = re.search(r'"timestamp"\s*:\s*"([0-9T:\-]{19})', line)
if m:
if first is None:
first = m.group(1)
last = m.group(1)
except OSError:
return None, None
def epoch(s):
try:
return time.mktime(time.strptime(s, "%Y-%m-%dT%H:%M:%S"))
except (ValueError, OverflowError):
return None
return epoch(first), epoch(last)
def wrap_inside(span, wrap_mtimes, lead=300, lag=900):
"""Did a wrap record get written inside this session's span?"""
a, b = span
if a is None or b is None:
return None # unreadable — not the same as 'no wrap'
return any(a - lead <= m <= b + lag for m in wrap_mtimes)
def sec_unwrapped():
"""
PENDING-S2's obligation, rebuilt on our own substrate.
A session that ends without /wrap-up writes no memory file, so `Last wrap`
— computed from mtime — silently reports the session BEFORE it, and the
thread below is inherited from the wrong session. The original proposal
detected this via a MemPalace Stop hook; that hook is retired, but the
failure it named is live. The transcripts are the only witness that a
session ran at all.
"""
proj = os.path.dirname(os.path.join(HOME, ".claude", "projects",
"-Users-davidglidden", "memory"))
try:
tx = [os.path.join(proj, f) for f in os.listdir(proj) if f.endswith(".jsonl")]
except OSError:
return None
now = time.time()
# The current session's transcript is being appended right now; the previous
# session's is the newest one that has gone quiet.
prior = [p for p in tx if now - os.path.getmtime(p) > 60]
if not prior:
return None
last = max(prior, key=os.path.getmtime)
verdict = wrap_inside(transcript_span(last),
[os.path.getmtime(w) for w in wrap_records()])
if verdict is None or verdict:
return None
ended = time.strftime("%b %d %H:%M", time.localtime(os.path.getmtime(last)))
return (f"⚠ PREVIOUS SESSION DID NOT WRAP (ended ~{ended}). The thread and "
f"question below are inherited from an OLDER session — treat them as "
f"possibly stale, and expect no record of what that session did.\n"
f" Not established: whether that session did work worth keeping. The "
f"transcript is on disk and can be read if the gap matters.")
def ruled_pendings(reviewed_text):
"""
The set of PENDING ids that rulings actually DISPOSE OF.
Resolved by the PENDING each ruling NAMES in its header, never by the REVIEWED
number. Those numbering sequences have drifted apart: REVIEWED-84 rules on
PENDING-87. Matching on the number alone suppressed PENDING-84 from the wake
digest on the very morning the steward's pulling thread pointed at it, and
closing that item later produced no visible change because it had never been
counted. Measured at the time: 9 suppressed, 8 correctly, 1 falsely.
A ruling whose header names no PENDING (REVIEWED-78, -81, -82 …) suppresses
nothing.
The correction runs in BOTH directions, and an earlier draft of this docstring
claimed otherwise — that it could only ever surface more, never fewer. That was
an overclaim, caught by the change proof rather than by reading. Measured against
the live files at the time of the fix: 3 items surfaced that had been falsely
hidden (PENDING-78, -81, -82 — like-numbered rulings exist, concerning other
matters), and 2 stopped being shown that were genuinely ruled (PENDING-87 by
REVIEWED-84, PENDING-88 by REVIEWED-85 — no REVIEWED-87 or -88 exists, so the
number-match had never suppressed them). Net 18 → 19 visible. Number-matching
was wrong in both directions; only subject-matching is right in either.
"""
return set(re.findall(r"^## REVIEWED-\S+\s*—\s*PENDING-(\S+?)\s*—", reviewed_text, re.M))
def sec_pending():
t = read(PENDING)
if t is None:
warn.append("PENDING.md unreadable")
return []
rt = read(REVIEWED) or ""
ruled = ruled_pendings(rt)
items = []
for h, ln in open_items(t):
m = re.match(r"PENDING-(\S+?)\s*—", h)
if m and m.group(1) in ruled:
continue
items.append((h, ln, tag_of(t, h)))
if not items:
warn.append("PENDING.md parsed but yielded no open items — check the parser")
return items
def sec_reviewed(n=3):
t = read(REVIEWED)
if t is None:
warn.append("REVIEWED.md unreadable")
return []
hs = re.findall(r"^## (REVIEWED-\S+ —.*)$", t, re.M)
out = []
for h in hs[-n:]:
m = re.search(re.escape("## " + h) + r".*?\n\*\*Decision:\*\*\s*(\w+)", t, re.S)
out.append((h[:88], m.group(1) if m else "?"))
return out
def sec_drift():
o = sh([sys.executable, DRIFT])
if o is None:
warn.append("drift check did not run")
return None
if "INSTRUMENT NOT VERIFIED" in o:
return "INSTRUMENT NOT VERIFIED — treat as unestablished"
m = re.search(r"(\d+) claim", o)
return m.group(1) if m else "0"
def sec_repos():
rows = []
for r in REPOS:
p = os.path.join(HOME, "_Dev", r)
if not os.path.isdir(os.path.join(p, ".git")):
continue
sb = sh(["git", "-C", p, "status", "-sb", "--porcelain"]) or ""
lines = sb.split("\n")
head = lines[0][3:] if lines and lines[0].startswith("##") else "?"
dirty = sum(1 for l in lines[1:] if l.strip())
subj = sh(["git", "-C", p, "log", "-1", "--format=%s"]) or "?"
rows.append((r, head, dirty, subj[:60]))
if not rows:
warn.append("no repos resolved")
return rows
def sec_thread(newest):
t = read(os.path.join(MEM, newest)) if newest else None
if t is None:
warn.append("session file unreadable — no thread or question")
return None, None
th = extract_anchor(t, "PULLING THREAD", 400)
q = extract_anchor(t, "LITERAL QUESTION", 900)
if th is None:
warn.append("no PULLING THREAD anchor in the last wrap")
if q is None:
warn.append("no LITERAL QUESTION anchor in the last wrap")
return th, q
# ---------- the .app brief ----------
# The jurist reads Claude.app's personal preferences and has NO filesystem access, so
# unlike CLAUDE.md that document cannot compute its state — it can only cache it. A cache
# drifts; the mitigations are to keep it small, generate rather than hand-write it, and
# date it so staleness is visible rather than misleading (Constitutional Constraint #4
# across a boundary we cannot check). This emits a paste-ready block and records when it
# was generated, so the wake can report the age.
BRIEF_PATH = os.path.join(D, "claude", "app-brief.md")
def trackers():
t = read(os.path.join(MEM, "MEMORY.md"))
if t is None:
warn.append("MEMORY.md unreadable — no workstreams for the brief")
return []
m = re.search(r"^## Canonical Workstream Trackers\s*$(.*?)^## ", t, re.S | re.M)
if not m:
warn.append("MEMORY.md has no 'Canonical Workstream Trackers' heading")
return []
def clip(s, n):
s = re.sub(r"\s+", " ", s).strip()
if len(s) <= n:
return s
return s[:n].rsplit(" ", 1)[0] + "…"
out = []
for line in m.group(1).split("\n"):
if not line.startswith("- "):
continue
body = re.sub(r"\[([^\]]+)\]\([^)]*\)", r"\1", line[2:]) # unlink
body = re.sub(r"[*`]", "", body)
name, _, gist = body.partition(" — ")
out.append((clip(name, 38), clip(gist, 72)))
return out
def commits_30d(path):
o = sh(["git", "-C", path, "log", "--since=30.days", "--oneline"])
return len(o.split("\n")) if o else 0
def brief(stamp):
"""Replaces ONLY the projects half of the .app preferences' §Standing Context.
The personal entries there (the orchestra conflict, the divorce, fraternal practice)
are steward-held, unverifiable from any substrate, and must never be generated."""
items = sec_pending()
o = [f"### Standing Context — Projects *(generated {stamp}; do not hand-edit)*",
"",
"Regenerate: `python3 ~/dotfiles/scripts/wake-digest.py --brief`. This is a snapshot,",
"not a live view — the reader of this document has no filesystem access, so it cannot",
"be computed here. If today is more than ~30 days after the date above, treat every",
"line as unverified rather than current. Personal standing context is kept separately",
"and by hand.", "",
"TRACKER INDEX (from MEMORY.md — what exists, not what is hot; see repo activity"
" below for that)"]
for n, tail in trackers():
o.append(f" {n:<38} {tail}")
o.append(f"\nOPEN AUTHORIZATION ITEMS ({len(items)}) — full text in ~/PENDING.md; "
"closed items in ~/PENDING-archive.md")
for h, _ln, tag in items:
o.append(f" {tag:<12} {h[:92]}")
o.append("\nLAST RULINGS")
for h, d in sec_reviewed(4):
o.append(f" {d:<10} {h}")
o.append("\nREPO ACTIVITY (commits, last 30 days)")
for r in REPOS:
p = os.path.join(HOME, "_Dev", r)
if os.path.isdir(os.path.join(p, ".git")):
o.append(f" {r:<24} {commits_30d(p):>4}")
o.append(f"\nGOVERNANCE DRIFT — ~/CLAUDE.md: {sec_drift()} substrate-contradicted claim(s)")
o.append("\n*(end generated block)*")
return "\n".join(o) + "\n"
def brief_age_days():
"""Age from the date written INSIDE the brief, never from mtime: git does not
preserve mtime, so a fresh clone would make a stale brief read as newly
generated — false freshness inside the freshness instrument."""
t = read(BRIEF_PATH)
if t is None:
return None
m = re.search(r"generated (\d{4})-(\d{2})-(\d{2})", t)
if not m:
warn.append("app-brief.md carries no 'generated <date>' stamp — age unknown")
return None
y, mo, d = (int(x) for x in m.groups())
return (time.time() - time.mktime((y, mo, d, 12, 0, 0, 0, 0, -1))) / 86400
# ---------- memory pointers ----------
#
# Provenance: 2026-07-28. Earned expensively. Six pointers in MEMORY.md /
# MEMORY-reference.md pointed at files that all EXIST via paths that never
# resolved (two at depth 4, four at depth 5 — hand-counted, never checked).
# The wake canary detected this and FIRED FOUR TIMES over two days. Each firing
# was re-diagnosed by hand and dismissed as "known canary bug", and the banked
# remedy was to change the canary's path resolution — i.e. to make the
# instrument agree with the broken data and stop reporting a TRUE POSITIVE.
#
# The defect was never the pointers; it was that the alarm said only MISSING.
# An undifferentiated alarm has to be re-diagnosed on every firing, and the
# cheapest re-diagnosis is always "known bug". So this reports three outcomes,
# never one, and hands back the exact replacement string:
#
# OK resolves from the file's real location
# MIS-AUTHORED target exists, path as written does not reach it -> fix given
# DEAD no file of that name anywhere -> a decision
#
# Relative pointers are resolved against the file's REAL directory, because the
# memory dir has two live addresses (~/.claude/projects/…/memory is a symlink to
# ~/dotfiles/claude/memory) and a relative escape is only ever valid from one of
# them. That ambiguity is structural: any pointer LEAVING the directory must be
# home-anchored (`~/…`) to be correct via both routes.
#
# Home-anchored, NOT absolute — steward's correction, same session. `/Users/
# davidglidden/…` resolves today and hardcodes this machine into a dotfiles repo
# whose whole purpose is surviving a machine change. `expanduser` is already the
# idiom three lines into this file, and `~/_Dev/…` is already the idiom in the
# memory files' own prose (17 uses). Absolute still VALIDATES here — it is not
# wrong, only unportable — but it is counted and reported so it cannot re-enter
# silently.
#
# 2026-08-10 — three completions, all earned before this code existed.
#
# (1) CODE SPANS. The matcher read `](file.md)` inside backticks as a pointer,
# so on 2026-08-09 it reported exactly one DEAD pointer: the link pattern
# written inside MEMORY.md's own SPECIFICATION OF THIS CANARY. An
# instrument that flags its own documentation flags it every wake, forever,
# and the real signal drowns in a permanent known-false line — the same
# "known canary bug" dismissal the block above was written to end, arriving
# by a second route. Code spans and fenced blocks are blanked before
# matching, offsets preserved so every reported line number stays true.
# Inline spans may not cross a newline: a stray unmatched backtick must
# bound its damage to one line rather than blanking the rest of the file
# and HIDING dead pointers. An unterminated fence simply does not match,
# so the failure direction is over-reporting, never silence.
#
# (2) WIKILINKS. `reference-verification-ladder.md` has specified the canary as
# covering "every `](file.md)` and `[[wikilink]]`" since 2026-07-06; only
# the first half was ever built. Reported as UNWRITTEN, not DEAD, because
# ~/CLAUDE.md rules a wikilink with no file yet to be legitimate — it marks
# something worth writing. A permitted forward-reference and a broken index
# pointer are different findings and must not share a word.
#
# (3) PRE-EXISTING vs NEWLY BROKEN. The 2026-08-09 spec asked for it. Derived
# from git, NOT from a stored prior run: a state file would make this the
# one cached section in a digest whose governing property is that it is
# computed. Where git cannot answer — a target outside this repo — it says
# so rather than guessing.
NONPORTABLE_RE = re.compile(r"^/(Users|home)/")
MEM_INDEXES = ["MEMORY.md", "MEMORY-reference.md"]
POINTER_RE = re.compile(r"\]\(([^)\s]+\.md)\)")
WIKILINK_RE = re.compile(r"\[\[([^\[\]|#\n]+?)(?:#[^\[\]|\n]*)?(?:\|[^\[\]\n]*)?\]\]")
FENCE_RE = re.compile(r"^(?P<f>`{3,}|~{3,})[^\n]*\n.*?^(?P=f)[^\n]*$", re.M | re.S)
INLINE_CODE_RE = re.compile(r"`+[^`\n]*`+")
def blank_code(text):
"""Blank code-span and fenced-block CONTENT, preserving offsets and newlines.
Preserving offsets is the point: every line number this module reports is
computed from the blanked text, so it must still address the real line.
"""
def blank(m):
return "".join("\n" if c == "\n" else " " for c in m.group(0))
return INLINE_CODE_RE.sub(blank, FENCE_RE.sub(blank, text))
def find_basename(name):
"""Bounded search for a file of this basename. None => genuinely dead."""
local = os.path.join(os.path.realpath(MEM), name)
if os.path.exists(local):
return local
hit = sh(["find", os.path.join(HOME, "_Dev"), "-maxdepth", "6",
"-name", name, "-not", "-path", "*/node_modules/*",
"-not", "-path", "*/.git/*"], timeout=8)
return hit.splitlines()[0] if hit else None
def resolve(raw, base):
"""Home-anchored, absolute, or relative-to-the-file's-real-directory."""
if raw.startswith("~"):
return os.path.expanduser(raw)
if os.path.isabs(raw):
return raw
return os.path.join(base, raw)
def classify_pointers(text, base):
"""(n_checked, mis_authored, dead, nonportable) — outcomes, never one word."""
n, mis, dead, nonport = 0, [], [], []
text = blank_code(text)
for m in POINTER_RE.finditer(text):
raw = m.group(1)
if raw.startswith(("http://", "https://", "mailto:")):
continue
n += 1
line = text.count("\n", 0, m.start()) + 1
if NONPORTABLE_RE.match(raw):
nonport.append((line, raw))
if os.path.exists(resolve(raw, base)):
continue
found = find_basename(os.path.basename(raw))
if found:
home = os.path.realpath(HOME)
real = os.path.realpath(found)
fix = "~" + real[len(home):] if real.startswith(home + os.sep) else real
mis.append((line, raw, fix))
else:
dead.append((line, raw, None))
return n, mis, dead, nonport
def near_slugs(slug, memdir):
"""Memory files this wikilink was plainly REACHING FOR. Prefix-family only.
`[[trust-prior-pass-frame]]` wants `feedback-trust-prior-pass-frame.md`; the
slug is the tail of the real stem after a hyphen. Matching on that boundary
(never on a bare substring) keeps `[[arc]]` from claiming every file with
"arc" inside a word.
"""
out = []
try:
stems = [f[:-3] for f in os.listdir(memdir) if f.endswith(".md")]
except OSError:
return out
for stem in stems:
if stem.endswith("-" + slug) or slug.endswith("-" + stem):
out.append(stem)
return sorted(out)
def classify_wikilinks(text, memdir):
"""(n_checked, mis_authored, unwritten) — three outcomes, never one word.
The same discipline as the path pointers above, and for the same reason: the
first build of this function reported only UNWRITTEN, and its two live hits
were both `[[trust-prior-pass-frame]]`, whose file EXISTS as
`feedback-trust-prior-pass-frame.md`. One word would have sent a fixable
typo to the wake every morning wearing the label "nothing to do here".
UNWRITTEN, never DEAD, for the genuine misses: ~/CLAUDE.md rules that a
`[[name]]` matching no file yet "is fine — it marks something worth writing
later, not an error." A permitted forward-reference is not a defect.
"""
n, mis, unwritten = 0, [], []
text = blank_code(text)
for m in WIKILINK_RE.finditer(text):
slug = m.group(1).strip()
if not slug:
continue
n += 1
stem = slug[:-3] if slug.endswith(".md") else slug
if os.path.exists(os.path.join(memdir, stem + ".md")):
continue
line = text.count("\n", 0, m.start()) + 1
near = near_slugs(stem, memdir)
if near:
mis.append((line, slug, " | ".join(near)))
else:
unwritten.append((line, slug))
return n, mis, unwritten
# Breakage age is DERIVED from git, never from a stored prior run — a cache is
# the one thing this digest's governing property forbids. Two questions, both
# answerable at HEAD: was the pointer already written, and did its target
# already exist? Their four combinations are the four honest verdicts.
def breakage(raw, head_text, target_at_head):
"""PRE-EXISTING vs NEWLY BROKEN. `None` inputs mean 'git cannot say'."""
if head_text is None:
return "age unknown — no committed version of this index to compare"
if raw not in head_text:
return "NEW — this pointer was added since HEAD"
if target_at_head is True:
return "NEWLY BROKEN — target existed at HEAD and is gone now"
if target_at_head is False:
return "pre-existing — already broken at HEAD"
return "pre-existing at HEAD — target lives outside this repo, break date unknown"
def head_text_of(relpath):
"""The index as committed. None => never committed, or git unavailable."""
return sh(["git", "-C", D, "show", f"HEAD:{relpath}"])
def target_at_head(raw, memrel):
"""Did the pointer's target exist at HEAD? None where git cannot know."""
base = os.path.basename(raw)
if os.path.normpath(resolve(raw, os.path.realpath(MEM))) != os.path.normpath(
os.path.join(os.path.realpath(MEM), base)):
return None # leaves the memory dir — outside git's reach here
return sh(["git", "-C", D, "cat-file", "-e", f"HEAD:{memrel}/{base}"]) is not None
def sec_pointers():
total, mis, dead, nonport = 0, [], [], []
wtotal, wmis, unwritten = 0, [], []
memdir = os.path.realpath(MEM)
memrel = os.path.relpath(memdir, D)
for name in MEM_INDEXES:
path = os.path.realpath(os.path.join(MEM, name))
text = read(path)
if text is None:
warn.append(f"memory pointers: {name} unreadable")
continue
n, m_, d_, np_ = classify_pointers(text, os.path.dirname(path))
total += n
head = head_text_of(f"{memrel}/{name}")
mis += [(name,) + t for t in m_]
dead += [(name, ln, raw, breakage(raw, head, target_at_head(raw, memrel)))
for ln, raw, _ in d_]
nonport += [(name,) + t for t in np_]
wn, wm, wu = classify_wikilinks(text, memdir)
wtotal += wn
wmis += [(name,) + t for t in wm]
unwritten += [(name,) + t for t in wu]
return total, mis, dead, nonport, wtotal, wmis, unwritten
# ---------- stray maps ----------
#
# A "map" is an artifact the steward reads directly to orient (dashboard, corpus
# index, reading list). Provenance 2026-07-28: arc-current-state-2026-05-07.md
# was exactly that, lived only on the Desktop, and vanished with a tidy-up; a
# census then found four more in the same condition, zero copies anywhere. Maps
# now live in ~/dotfiles/maps and are symlinked back to the Desktop. This reports
# any Desktop .md that is NOT such a symlink — it never moves anything, because
# the Desktop is the steward's. See maps/README.md.
DESKTOP = os.path.join(HOME, "Desktop")
MAPS_DIR = os.path.join(D, "maps")
def is_homed(path, maps_root):
"""True iff path is a symlink resolving inside maps_root."""
if not os.path.islink(path):
return False
return os.path.realpath(path).startswith(os.path.realpath(maps_root) + os.sep)
def stray_maps():
try:
return [n for n in sorted(os.listdir(DESKTOP))
if n.endswith(".md") and not is_homed(os.path.join(DESKTOP, n), MAPS_DIR)]
except Exception:
warn.append("stray-map check: Desktop unreadable")
return []
# ---------- self-test ----------
def selftest():
ok = True
def chk(name, cond):
nonlocal ok
ok = ok and cond
print(f" [{'ok ' if cond else 'FAIL'}] {name}")
print("extractor controls (presence AND absence):")
chk("open_items finds a plain item",
open_items("## PENDING-1 — x") == [("PENDING-1 — x", 1)])
chk("open_items finds a non-numeric family [v1 splitter's blind spot]",
open_items("## PENDING-S2 — x\n## PENDING — ICP-19") and
len(open_items("## PENDING-S2 — x\n## PENDING — ICP-19")) == 2)
chk("open_items excludes CLOSED / COMPLETED",
open_items("## PENDING-1 — x (CLOSED)\n## COMPLETED — y") == [])
chk("open_items returns empty on empty input", open_items("") == [])
FENCED = "## PENDING-9 — real\nbody\n```markdown\n## REVIEWED-9 — a draft\n```\ntail"
chk("open_items ignores a header inside a fenced block [drafting-habit trigger]",
[h for h, _ in open_items(FENCED)] == ["PENDING-9 — real"])
chk("open_items DOES find that same header unfenced [positive control for the line above]",
[h for h, _ in open_items(FENCED.replace("```markdown\n", "").replace("```\n", ""))]
== ["PENDING-9 — real", "REVIEWED-9 — a draft"])
chk("item_spans runs a fenced item to the tail, not to the fenced header",
item_spans(FENCED)[0][2] == 7)
chk("tag_of reads a tag",
tag_of("## PENDING-9 — t\n**Date:** d\n**Tag:** [ESCALATE]\n", "PENDING-9 — t") == "[ESCALATE]")
chk("tag_of returns '' when absent", tag_of("## PENDING-9 — t\n", "PENDING-9 — t") == "")
chk("ruled_pendings resolves by the NAMED pending, not the REVIEWED number",
ruled_pendings("## REVIEWED-84 — PENDING-87 — Order attestation") == {"87"})
chk("ruled_pendings does NOT suppress the like-numbered item [the 2026-08-01 bug]",
"84" not in ruled_pendings("## REVIEWED-84 — PENDING-87 — Order attestation"))
chk("ruled_pendings ignores a ruling that names no PENDING",
ruled_pendings("## REVIEWED-82 — Read-only MCP server: eyes on the substrate") == set())
chk("ruled_pendings handles non-numeric families",
ruled_pendings("## REVIEWED-9 — PENDING-S2 — hook-aware deposit") == {"S2"})
chk("ruled_pendings returns empty on empty input [positive control]",
ruled_pendings("") == set())
chk("extract_anchor pulls verbatim",
extract_anchor("noise\n**PULLING THREAD:** do the thing.\n\nmore", "PULLING THREAD")
== "do the thing.")
chk("extract_anchor handles label-inside-bold (**A: content**)",
extract_anchor("**PULLING THREAD: do the thing.**\n\nmore", "PULLING THREAD")
== "do the thing.")
chk("extract_anchor drops a label remnant (**A for next-Claude:** content)",
extract_anchor("**LITERAL QUESTION for next-Claude:** Is it?\n\nx", "LITERAL QUESTION")
== "Is it?")
chk("extract_anchor skips YAML frontmatter",
extract_anchor('---\ndescription: "PULLING THREAD: the wrong one."\ntype: x\n---\n'
"**PULLING THREAD: the right one.**\n\nz", "PULLING THREAD")
== "the right one.")
chk("extract_anchor marks truncation",
extract_anchor("**A:** " + "w " * 200, "A", limit=50).endswith("…[truncated]"))
chk("extract_anchor returns None when the anchor is absent",
extract_anchor("no anchor here", "PULLING THREAD") is None)
MEMDIR = os.path.realpath(MEM)
TOUCH = "~/_Dev/studium-engine/docs/the-chamber-touchstone.md"
chk("classify_pointers accepts an in-directory relative pointer",
classify_pointers("see [x](MEMORY.md)", MEMDIR) == (1, [], [], []))
chk("classify_pointers accepts a HOME-ANCHORED pointer [the chosen form]",
classify_pointers(f"[t]({TOUCH})", MEMDIR) == (1, [], [], []))
chk("classify_pointers calls a real-target/bad-path pointer MIS-AUTHORED"
" [the 4-firing bug, as a regression control]",
(lambda r: r[0] == 1 and len(r[1]) == 1 and not r[2])(
classify_pointers(
"[t](../../../../_Dev/studium-engine/docs/the-chamber-touchstone.md)",
MEMDIR)))
chk("...and the fix it hands back is home-anchored, not machine-absolute",
classify_pointers("[t](../../../../_Dev/studium-engine/docs/"
"the-chamber-touchstone.md)", MEMDIR)[1][0][2] == TOUCH)
chk("classify_pointers calls a nowhere-target pointer DEAD [must NOT collapse"
" into mis-authored]",
(lambda r: r[0] == 1 and not r[1] and len(r[2]) == 1)(
classify_pointers("[x](no-such-file-anywhere-xyzzy-9931.md)", MEMDIR)))
chk("classify_pointers flags a machine-absolute pointer NON-PORTABLE even though"
" it resolves [prevents silent re-entry]",
(lambda r: len(r[3]) == 1 and not r[1] and not r[2])(
classify_pointers(f"[t]({os.path.expanduser(TOUCH)})", MEMDIR)))
chk("classify_pointers ignores http links",
classify_pointers("[x](https://a.md)", MEMDIR) == (0, [], [], []))
chk("classify_pointers returns clean on empty input",
classify_pointers("", MEMDIR) == (0, [], [], []))
NOWHERE = "no-such-file-anywhere-xyzzy-9931.md"
print("\ncode spans [2026-08-09: the canary flagged its own specification]:")
chk("a pointer INSIDE a code span is not a pointer",
classify_pointers(f"`[y]({NOWHERE})`", MEMDIR) == (0, [], [], []))
chk("...and the SAME pointer outside one still fires [direction control —"
" blanking must not swallow everything]",
classify_pointers(f"[y]({NOWHERE})", MEMDIR)[0] == 1)
chk("MEMORY.md's own canary spec is silent [the exact 2026-08-09 false positive]",
(lambda t: classify_pointers(t, MEMDIR) == (0, [], [], [])
and classify_wikilinks(t, MEMDIR) == (0, [], []))(
"→ `~/dotfiles/scripts/`, invoked not described; `](file.md)` +"
" `[[wikilink]]` over both memory indexes"))
chk("blanking preserves LINE NUMBERS [offsets kept, not deleted]",
classify_pointers(f"`](x.md)`\n[y]({NOWHERE})\n", MEMDIR)[2][0][0] == 2)
chk("a fenced block is blanked",
classify_pointers(f"```\n[y]({NOWHERE})\n```\n", MEMDIR) == (0, [], [], []))
chk("a STRAY unmatched backtick blanks nothing [damage bounded to one line;"
" a greedy matcher would HIDE dead pointers]",
classify_pointers(f"a ` b\n[y]({NOWHERE})\n", MEMDIR)[0] == 1)
chk("an UNTERMINATED fence blanks nothing [fails toward reporting, not silence]",
classify_pointers(f"```\n[y]({NOWHERE})\n", MEMDIR)[0] == 1)
print("\nwikilinks [the ladder specified them 2026-07-06; never built until now]:")
chk("a wikilink whose file exists resolves",
classify_wikilinks("see [[MEMORY]]", MEMDIR) == (1, [], []))
chk("a wikilink with no file is UNWRITTEN, not dead [~/CLAUDE.md rules it legitimate]",
(lambda r: r[0] == 1 and not r[1] and len(r[2]) == 1)(
classify_wikilinks("see [[no-such-memory-xyzzy-9931]]", MEMDIR)))
chk("a NEAR-MISS slug is MIS-AUTHORED with the real file handed back"
" [live: the two hits the one-word version mislabelled]",
(lambda r: len(r[1]) == 1 and not r[2]
and r[1][0][2] == "feedback-trust-prior-pass-frame")(
classify_wikilinks("[[trust-prior-pass-frame]]", MEMDIR)))
chk("...and a genuine miss does NOT collapse into mis-authored [direction control]",
not classify_wikilinks("[[no-such-memory-xyzzy-9931]]", MEMDIR)[1])
chk("near_slugs matches on the HYPHEN boundary, not bare substring"
" [else [[arc]] would claim every file with 'arc' in a word]",
all(s.endswith("-arc") or "arc".endswith("-" + s) for s in near_slugs("arc", MEMDIR)))
chk("an alias and a heading are stripped down to the slug",
classify_wikilinks("[[MEMORY#Index|the index]]", MEMDIR) == (1, [], []))
chk("a wikilink inside a code span is not a wikilink",
classify_wikilinks("`[[no-such-memory-xyzzy-9931]]`", MEMDIR) == (0, [], []))
chk("classify_wikilinks returns clean on empty input [positive control]",
classify_wikilinks("", MEMDIR) == (0, [], []))
print("\nbreakage age [derived from git — this digest may not cache]:")
chk("a pointer absent at HEAD is NEW",
breakage("x.md", "nothing here", False).startswith("NEW"))
chk("a pointer present at HEAD whose target was there too is NEWLY BROKEN",
breakage("x.md", "[a](x.md)", True).startswith("NEWLY BROKEN"))
chk("a pointer present at HEAD whose target was already gone is pre-existing",
breakage("x.md", "[a](x.md)", False).startswith("pre-existing"))
chk("an out-of-repo target says the break date is unknown [honest degradation]",
"unknown" in breakage("x.md", "[a](x.md)", None))
chk("no committed index reads as UNKNOWN, never as pre-existing"
" [negative control — absence of evidence]",
breakage("x.md", None, None).startswith("age unknown"))
chk("is_homed TRUE for a Desktop map symlinked into maps/ [live substrate]",
is_homed(os.path.join(DESKTOP, "corpus-index-2026-07-26.md"), MAPS_DIR))
chk("is_homed FALSE for a regular file [negative control — must not pass everything]",
not is_homed(os.path.join(MAPS_DIR, "README.md"), MAPS_DIR))
chk("is_homed FALSE for a symlink pointing OUTSIDE maps/",
not is_homed(os.path.join(HOME, ".claude"), MAPS_DIR))
print("\nunwrapped-session detector [PENDING-S2's obligation, our substrate]:")
chk("wrap_inside TRUE when a wrap falls inside the span",
wrap_inside((1000.0, 2000.0), [1500.0]))
chk("wrap_inside TRUE for a wrap just after the last write [wraps land near the end]",
wrap_inside((1000.0, 2000.0), [2400.0]))
chk("wrap_inside FALSE when every wrap is outside [negative control]",
not wrap_inside((1000.0, 2000.0), [500.0, 5000.0]))
chk("wrap_inside returns None on an unreadable span [must NOT read as 'no wrap']",
wrap_inside((None, None), [1500.0]) is None)
# Discrimination on REAL sessions: the detector must return BOTH verdicts over
# the actual transcript history. One verdict everywhere = it discriminates nothing.
_proj = os.path.dirname(MEM)
_tx = sorted((os.path.join(_proj, f) for f in os.listdir(_proj)
if f.endswith(".jsonl")), key=os.path.getmtime)[-14:-1]
_wm = [os.path.getmtime(w) for w in wrap_records()]
_v = [wrap_inside(transcript_span(p), _wm) for p in _tx]
chk(f"real sessions read as WRAPPED [{sum(1 for x in _v if x is True)} of {len(_v)}]",
any(x is True for x in _v))
chk(f"real sessions read as UNWRAPPED [{sum(1 for x in _v if x is False)} of {len(_v)}]"
" — the negative instance; without one the detector is unproven",
any(x is False for x in _v))
print("\nlive substrate:")
chk("PENDING.md readable", read(PENDING) is not None)
chk("REVIEWED.md readable", read(REVIEWED) is not None)
chk("drift check runs", sec_drift() is not None)
chk("memory dir resolves", os.path.isdir(MEM))
print("\nSELFTEST", "PASS" if ok else "FAIL")
return 0 if ok else 2
# ---------- render ----------
def main():
newest, span = sec_pause()
th, q = sec_thread(newest)
items, revs, drift, repos = sec_pending(), sec_reviewed(), sec_drift(), sec_repos()
o = ["=== WAKE DIGEST (computed now — not a stored snapshot) ==="]
o.append(f"Last wrap: {span} ago ({newest})" if span else "Last wrap: UNKNOWN")
unwrapped = sec_unwrapped()
if unwrapped:
o.append(unwrapped)
if th:
o.append(f"\nPULLING THREAD — {th}")
if q:
o.append(f"\nOPEN QUESTION — {q}")
o.append(f"\nOPEN AUTHORIZATION ITEMS ({len(items)}) — bodies in ~/PENDING.md at the line given;"
" closed items are in ~/PENDING-archive.md")
for h, ln, tag in items:
o.append(f" L{ln:<5} {tag:<12} {h[:78]}")
if revs:
o.append("\nLAST RULINGS")
for h, d in revs:
o.append(f" {d:<10} {h}")
o.append(f"\nGOVERNANCE DRIFT — ~/CLAUDE.md: {drift} substrate-contradicted claim(s)"
" (detection only; correction needs [ESCALATE])")
ptot, pmis, pdead, pnp, wtot, wmis, wun = sec_pointers()
o.append(f"\nMEMORY POINTERS — {ptot} checked · {len(pmis)} mis-authored · {len(pdead)} dead"
+ (f" · {len(pnp)} non-portable" if pnp else "")
+ f" | {wtot} wikilinks · {len(wmis)} mis-authored · {len(wun)} unwritten")
for f, ln, raw, fix in pmis:
o.append(f" MIS-AUTHORED {f}:{ln} {raw}")
o.append(f" → target exists; replace with: {fix}")
for f, ln, raw, age in pdead:
o.append(f" DEAD {f}:{ln} {raw} (no file of that name in ~/_Dev or the memory dir)")
o.append(f" → {age}")
for f, ln, raw in pnp:
o.append(f" NON-PORTABLE {f}:{ln} {raw} (resolves, but hardcodes this machine)")
for f, ln, slug, near in wmis:
o.append(f" MIS-AUTHORED {f}:{ln} [[{slug}]]")
o.append(f" → the file exists; replace with: [[{near}]]")
for f, ln, slug in wun:
o.append(f" UNWRITTEN {f}:{ln} [[{slug}]] (permitted forward-reference, not an error)")
strays = stray_maps()
if strays:
o.append(f"\nSTRAY MAPS — {len(strays)} Desktop .md with no tracked copy in"
" ~/dotfiles/maps/ (one tidy-up from gone; see maps/README.md)")
for s in strays:
o.append(f" {s}")
age = brief_age_days()
if age is None:
o.append("\n.APP BRIEF — never generated. The jurist's §Standing Context cannot be"
" checked from here; run `wake-digest.py --brief` and paste it.")
elif age > 30:
o.append(f"\n.APP BRIEF — last GENERATED {age:.0f} days ago. This tracks generation,"
" not pasting: it is a lower bound on the jurist's staleness, never a"
" guarantee of freshness.")
if repos:
o.append("\nREPOS")
for r, head, dirty, subj in repos:
flag = f" ~{dirty} dirty" if dirty else ""
o.append(f" {r:<24} {head:<34}{flag} {subj}")
if warn:
o.append("\n⚠ DIGEST DEGRADED — these sections could not be computed:")
for w in warn:
o.append(f" - {w}")
o.append("=== END WAKE DIGEST ===")
print("\n".join(o))
def emit_brief():
stamp = sh(["date", "+%Y-%m-%d"]) or "date-unavailable"
text = brief(stamp)
os.makedirs(os.path.dirname(BRIEF_PATH), exist_ok=True)
open(BRIEF_PATH, "w", encoding="utf-8").write(text)
print(text)
print(f"[written to {BRIEF_PATH} — the wake reports this file's age]", file=sys.stderr)
if warn:
print("⚠ brief degraded: " + "; ".join(warn), file=sys.stderr)
if __name__ == "__main__":
try:
if "--brief" in sys.argv:
sys.exit(emit_brief())
sys.exit(selftest() if "--selftest" in sys.argv else main())
except Exception as e: # never break session start
print(f"=== WAKE DIGEST UNAVAILABLE — {type(e).__name__}: {e} ===")