---
name: feedback-checkable-question-over-self-authored-corpus
description: "A question is not checkable merely because it counts events in a record. If the record is authored by the party being measured, it is self-report with extra steps — and the checkable FORM disguises that."
metadata:
  node_type: memory
  type: feedback
  modified: 2026-08-20T21:51:44Z
---

# A checkable question over a self-authored corpus is self-report with extra steps

`/wrap-up` §1 requires the literal question for next-Claude to prefer **the checkable form over
the self-report form**, because direct self-report is *"the most contaminated form of inquiry
available."* The rule is right. **The trap is that a question can satisfy its letter and none of
its purpose.**

**The instance, 2026-08-20.** The wrap left this question: *"across the last ~15 sessions, when a
claim about a document turned out wrong, how many were caught by a party disagreeing versus by
someone opening the document?"* It looks checkable — it counts events in a written record rather
than asking the executor to introspect. It was posed while quoting the very discipline it fails.

**Why it fails.** The record is the Symmetria ledgers: 15 files, 244 entries, 79 correction-shaped
in the window — **every one of them written by the executor, about the executor's own errors.**
Counting them measures what the executor chose to record and how it chose to characterise the
catch. Under-recording catches made *on* it and over-recording them are both available, and neither
is detectable from inside. The counting step is mechanical; **the corpus is testimony.**

Two further failures showed up on execution and are worth carrying:

- **Keyword extraction over prose testimony does not rescue it.** 34% unclassifiable, and the
  "caught by a party" bucket filled with entries that merely *mention* the jurist or steward. A
  number off that classifier is worse than no number, because it launders an interpretive judgement
  into a percentage.
- **The narrower the query, the less it depends on the author's framing.** *"Does any entry name a
  disclosed limit as the cause of a correction?"* returned answers the entries state in words —
  far less interpretive than *"who caught it"*, which requires reading intent.

**The test to apply before leaving a question.** Not *"does this count something?"* but:

> **Who wrote the record this question reads, and would a different author have written it
> differently?**

If the answer is *the party being measured*, the question is self-report however mechanical the
counting. Either find a corpus with a different author (git history, the substrate itself, a
jurist's or steward's own words), or **narrow the query until it turns on what the entries
literally say rather than on what they meant.**

Kin to [[feedback-census-by-mechanism-not-proxy]] (census by running the real pipeline) and to
[[feedback-checkable-claim-surfaces-bugs]]. The difference this one adds: those concern whether the
*instrument* is real; this concerns whether the *corpus* is independent of the party asking.