[FIX] fool trial 03 VOID; degraded-guard rebuilt with a positive control
Trial 03 ran and produced nothing gradeable. Recorded as VOID rather than
omitted, because an absent row reads as a trial not attempted.
Two independent failures, both found by reading the output, neither by a check,
and every check passed:
1. The harness certified a run with no answer. Qwen emitted its scratchpad as
plain prose ('Here's a thinking process:', zero <think> tags), so the tag
regex reported reasoning_present:false and recorded all 2,944 words of
deliberation as the ANSWER; the token ceiling then cut it off mid-sentence
before the answer began. degraded:null. The guard tested the STRING for
emptiness while its field claimed a property of the RESULT — which is the
previous session's open question, answered by the instrument built to audit
instruments. Trial 02 had listed the inline-scratchpad problem as Open; the
harness closed it assuming inline meant tagged.
2. Worse: the design forbade the region it was measuring. The self-exemption
axis lives in Part VII; the anti-echo constraint added in trial 02 tells the
reader to skip author-named limitations, and the scratchpad shows the model
reaching Part VII and leaving it, citing that constraint. Silence about
self-reference is indistinguishable from obedience. The axis was unmeasurable
by construction, independent of the truncation. Trial 02's fix and trial 03's
document were each sound alone; their interaction was not.
Guard now reports every degradation, not the first: empty answer, untagged
scratchpad, and token-ceiling truncation. reasoning_present renamed
think_tag_found — it was a claim about a regex wearing the name of a claim about
the model. test_degraded_guard.py is a positive control that runs against the
actual trial-03 artefact, not a synthetic one; it caught a false positive in the
first version of my own guard (a bare 'okay' matched a legitimate sentence).
The false-positive control STILL has never been run. Two attempts, two unrelated
causes — the obstacle is the instrument and the design, not the model.
This commit is contained in:
@@ -22,6 +22,14 @@
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 01 | 2026-08-01 | PENDING-88 skill-harvest FIX lane | Qwen 3.6 35B-A3B 8bit | MISS | **MET ×2** | n/a | untested | **missed** (narrower test is less safe) |
|
||||
| 02 | 2026-08-02 | order-attestation (2026-07-29) | Qwen 3.6 35B-A3B 8bit | MISS | **MET ×2** | avoided | untested | **missed** (independence axis) |
|
||||
| 03 | 2026-08-02 | differently-biased-checkers (2026-08-01) | Qwen 3.6 35B-A3B 8bit | **VOID** | **VOID** | n/a | **still untested** | n/a |
|
||||
|
||||
**Trial 03 is VOID and is entered as VOID rather than omitted** — an absent row would read as a trial not attempted. Write-up: `fool-trial-03-2026-08-02.md`. Two independent failures, both found by reading the output, neither by a check:
|
||||
|
||||
1. **The harness certified a run with no answer.** Qwen emitted an untagged scratchpad (`"Here's a thinking process:"`, zero `<think>` tags), so the tag regex reported `reasoning_present: false` and recorded all 2,944 words of deliberation as the *answer*; the token ceiling then cut it off mid-sentence before the answer began. `degraded: null`. The guard tested the string for emptiness while its field claimed the result was sound. Trial 02's write-up had listed the inline-scratchpad problem as **Open**; the harness closed it on the assumption that inline meant `<think>`-tagged.
|
||||
2. **The design forbade the region it was measuring.** The self-exemption axis lives in Part VII; the anti-echo constraint — added in trial 02 — instructs the reader to skip author-named limitations, and the scratchpad shows the model reaching Part VII and leaving it, *citing that constraint*. Silence on self-reference is therefore indistinguishable from obedience. **The axis was unmeasurable by construction**, independent of the truncation.
|
||||
|
||||
Trial 02's fix and trial 03's document were each sound alone; their interaction was not.
|
||||
|
||||
## What the log shows so far
|
||||
|
||||
@@ -47,6 +55,8 @@ The v1 Chamber (June–July 2025) ran written work past **two frontier models of
|
||||
|
||||
**No false-positive control has ever been run.** Every trial to date used a document with real weaknesses. The claim that the model will say *"nothing found"* on a sound document is **untested** — trial 02's apparent restraint was an artifact of a disabled reasoning mode. Until a clean document is run, the finding-rate cannot be distinguished from a production-rate.
|
||||
|
||||
*Still true after trial 03, which was attempted and voided (2026-08-02). Two attempts have now failed for two unrelated reasons, which is itself the finding: the obstacle is not the model but the instrument and the design around it.*
|
||||
|
||||
## Grading caveat, standing
|
||||
|
||||
Every grade above was assigned by the executor, whose own errors are among those being graded, and whose reading of what counts as "real" is the reading under test. The findings are individually checkable; the *grades* are not independent.
|
||||
|
||||
Reference in New Issue
Block a user