[FIX] fool trial 03 VOID; degraded-guard rebuilt with a positive control
Trial 03 ran and produced nothing gradeable. Recorded as VOID rather than
omitted, because an absent row reads as a trial not attempted.
Two independent failures, both found by reading the output, neither by a check,
and every check passed:
1. The harness certified a run with no answer. Qwen emitted its scratchpad as
plain prose ('Here's a thinking process:', zero <think> tags), so the tag
regex reported reasoning_present:false and recorded all 2,944 words of
deliberation as the ANSWER; the token ceiling then cut it off mid-sentence
before the answer began. degraded:null. The guard tested the STRING for
emptiness while its field claimed a property of the RESULT — which is the
previous session's open question, answered by the instrument built to audit
instruments. Trial 02 had listed the inline-scratchpad problem as Open; the
harness closed it assuming inline meant tagged.
2. Worse: the design forbade the region it was measuring. The self-exemption
axis lives in Part VII; the anti-echo constraint added in trial 02 tells the
reader to skip author-named limitations, and the scratchpad shows the model
reaching Part VII and leaving it, citing that constraint. Silence about
self-reference is indistinguishable from obedience. The axis was unmeasurable
by construction, independent of the truncation. Trial 02's fix and trial 03's
document were each sound alone; their interaction was not.
Guard now reports every degradation, not the first: empty answer, untagged
scratchpad, and token-ceiling truncation. reasoning_present renamed
think_tag_found — it was a claim about a regex wearing the name of a claim about
the model. test_degraded_guard.py is a positive control that runs against the
actual trial-03 artefact, not a synthetic one; it caught a false positive in the
first version of my own guard (a bare 'okay' matched a legitimate sentence).
The false-positive control STILL has never been run. Two attempts, two unrelated
causes — the obstacle is the instrument and the design, not the model.
This commit is contained in:
@@ -73,9 +73,15 @@ grade will not be independent.
|
||||
|
||||
## Addendum, written DURING the run and BEFORE any output was seen
|
||||
|
||||
*(Run launched 2026-08-02 ~13:0x; model still loading; the output file was empty when
|
||||
each item below was written. Recorded here rather than in the write-up precisely because
|
||||
its whole value is that it precedes the result.)*
|
||||
*(Run launched 2026-08-02 mid-afternoon; the model was still loading and the output file was
|
||||
verifiably empty when each item below was written. Recorded here rather than in the write-up
|
||||
precisely because its whole value is that it precedes the result — so the claim is committed
|
||||
as `b678d2f`, whose timestamp is checkable, rather than asserted in prose. The run's own
|
||||
`started_utc` in the run record is the other half of the ordering.*
|
||||
|
||||
*A wrong clock-time — "~13:0x" — stood in this line in `b678d2f`. It was four hours off, in a
|
||||
document whose entire load-bearing property is its timestamps. Corrected here rather than
|
||||
quietly, because the correction is the sort of thing this file exists to make visible.)*
|
||||
|
||||
**1. The "unruled" premise above expired 32 minutes after it was written.** It was true at
|
||||
11:41. At **12:13** the steward placed **REVIEWED-86**, design-gating this doctrine with two
|
||||
@@ -116,6 +122,40 @@ Second: the harness now hashes *itself* into the run record, because `git_revisi
|
||||
null whenever the harness runs outside its repository — which is always, since it must run on
|
||||
the machine holding the model.
|
||||
|
||||
## Result
|
||||
## Result — written AFTER the run, and marked as such
|
||||
|
||||
*(To be filled after the run. Empty until then — deliberately.)*
|
||||
**VOID.** Not STRONG HIT, not EXEMPTION SIGNAL, not NULL, not ECHO. The trial did not
|
||||
produce a gradeable output, and its axis could not have been measured even if it had.
|
||||
Full write-up: `../fool-trial-03-2026-08-02.md`. Run record:
|
||||
`runs/trial-03-20260802T144136Z.*`.
|
||||
|
||||
1. **No answer was produced.** Qwen emitted an untagged scratchpad and exhausted the
|
||||
4,096-token ceiling before beginning its answer. The harness recorded
|
||||
`degraded: null` — it tested the string for emptiness while the field claimed the
|
||||
result was sound. Fixed this session, with a positive control that runs against the
|
||||
actual artefact (`test_degraded_guard.py`).
|
||||
|
||||
2. **The axis was unmeasurable by construction, and this is the design's fault, not the
|
||||
run's.** The self-exemption signal lives in Part VII; the prompt's anti-echo constraint
|
||||
tells the reader to skip author-named limitations. The scratchpad shows the model
|
||||
reaching Part VII and leaving it, citing that constraint. Silence about self-reference
|
||||
is therefore indistinguishable from obedience.
|
||||
|
||||
**The pre-registration above did not catch this, and the reason is worth recording:
|
||||
it reasoned about the document and about the grading, and never about the prompt
|
||||
already sitting in the file.** The PROVENANCE note warned that the prompt was
|
||||
reconstructed and that trial 03 was not a one-variable step — and the warning was
|
||||
read as a caveat on *comparability* rather than as a reason to re-read what the
|
||||
prompt instructs. The ladder's own rule covers it: re-run verification at the scope
|
||||
of the extension.
|
||||
|
||||
**Ground truth (a)–(e) was not revised, and was not scored** — there is no valid output to
|
||||
score. The comparison against REVIEWED-86 is therefore **not performed**; it waits for a
|
||||
valid run.
|
||||
|
||||
**The pre-run addendum earned its keep.** Every item in it held up, and item 2 — the
|
||||
(a)/anti-echo collision, resolved against my own convenience before output existed — was
|
||||
the thread that led to failure 2. Having already ruled that the anti-echo clause excluded
|
||||
part of ground truth (a), the question *what else does it exclude?* was available. It was
|
||||
not asked until the output forced it, which is the honest limit on how much credit the
|
||||
addendum deserves.
|
||||
|
||||
Reference in New Issue
Block a user