Filed in trial 04's tool review, now closed. The guard reported UNTAGGED
SCRATCHPAD ... "Do not grade this as the checker's findings" for both of the two
situations it can see, and they are opposite:
trial 03 — deliberation that ran into the CEILING. No answer ever existed. VOID,
and the absence of findings is NOT restraint.
trial 04 — deliberation that COMPLETED. The answer follows the scratchpad in the
same file. Perfectly gradeable once extracted. NOT void.
Collapsing them would have thrown away six good runs; not distinguishing them
would have graded trial 03's silence as restraint. The guard now branches on
hit_token_ceiling and says which case it is.
Controls added for all four shapes, including the two the trials actually
produced and a clean answer that merely hit the ceiling — truncation is reported
separately and is not a scratchpad problem.
The guard does NOT auto-extract the embedded answer. A heuristic split would be a
new failure mode in the instrument whose entire job is to not silently mis-report
what it has. It flags; a person extracts.
Trial 03 ran and produced nothing gradeable. Recorded as VOID rather than
omitted, because an absent row reads as a trial not attempted.
Two independent failures, both found by reading the output, neither by a check,
and every check passed:
1. The harness certified a run with no answer. Qwen emitted its scratchpad as
plain prose ('Here's a thinking process:', zero <think> tags), so the tag
regex reported reasoning_present:false and recorded all 2,944 words of
deliberation as the ANSWER; the token ceiling then cut it off mid-sentence
before the answer began. degraded:null. The guard tested the STRING for
emptiness while its field claimed a property of the RESULT — which is the
previous session's open question, answered by the instrument built to audit
instruments. Trial 02 had listed the inline-scratchpad problem as Open; the
harness closed it assuming inline meant tagged.
2. Worse: the design forbade the region it was measuring. The self-exemption
axis lives in Part VII; the anti-echo constraint added in trial 02 tells the
reader to skip author-named limitations, and the scratchpad shows the model
reaching Part VII and leaving it, citing that constraint. Silence about
self-reference is indistinguishable from obedience. The axis was unmeasurable
by construction, independent of the truncation. Trial 02's fix and trial 03's
document were each sound alone; their interaction was not.
Guard now reports every degradation, not the first: empty answer, untagged
scratchpad, and token-ceiling truncation. reasoning_present renamed
think_tag_found — it was a claim about a regex wearing the name of a claim about
the model. test_degraded_guard.py is a positive control that runs against the
actual trial-03 artefact, not a synthetic one; it caught a false positive in the
first version of my own guard (a bare 'okay' matched a legitimate sentence).
The false-positive control STILL has never been run. Two attempts, two unrelated
causes — the obstacle is the instrument and the design, not the model.
David F Glidden