[FIX] fool trial 03 VOID; degraded-guard rebuilt with a positive control

Trial 03 ran and produced nothing gradeable. Recorded as VOID rather than
omitted, because an absent row reads as a trial not attempted.

Two independent failures, both found by reading the output, neither by a check,
and every check passed:

1. The harness certified a run with no answer. Qwen emitted its scratchpad as
   plain prose ('Here's a thinking process:', zero <think> tags), so the tag
   regex reported reasoning_present:false and recorded all 2,944 words of
   deliberation as the ANSWER; the token ceiling then cut it off mid-sentence
   before the answer began. degraded:null. The guard tested the STRING for
   emptiness while its field claimed a property of the RESULT — which is the
   previous session's open question, answered by the instrument built to audit
   instruments. Trial 02 had listed the inline-scratchpad problem as Open; the
   harness closed it assuming inline meant tagged.

2. Worse: the design forbade the region it was measuring. The self-exemption
   axis lives in Part VII; the anti-echo constraint added in trial 02 tells the
   reader to skip author-named limitations, and the scratchpad shows the model
   reaching Part VII and leaving it, citing that constraint. Silence about
   self-reference is indistinguishable from obedience. The axis was unmeasurable
   by construction, independent of the truncation. Trial 02's fix and trial 03's
   document were each sound alone; their interaction was not.

Guard now reports every degradation, not the first: empty answer, untagged
scratchpad, and token-ceiling truncation. reasoning_present renamed
think_tag_found — it was a claim about a regex wearing the name of a claim about
the model. test_degraded_guard.py is a positive control that runs against the
actual trial-03 artefact, not a synthetic one; it caught a false positive in the
first version of my own guard (a bare 'okay' matched a legitimate sentence).

The false-positive control STILL has never been run. Two attempts, two unrelated
causes — the obstacle is the instrument and the design, not the model.
This commit is contained in:
David F Glidden
2026-08-02 16:50:59 +02:00
parent b678d2f57b
commit eda11e559b
8 changed files with 608 additions and 15 deletions
+10
View File
@@ -22,6 +22,14 @@
|---|---|---|---|---|---|---|---|---|
| 01 | 2026-08-01 | PENDING-88 skill-harvest FIX lane | Qwen 3.6 35B-A3B 8bit | MISS | **MET ×2** | n/a | untested | **missed** (narrower test is less safe) |
| 02 | 2026-08-02 | order-attestation (2026-07-29) | Qwen 3.6 35B-A3B 8bit | MISS | **MET ×2** | avoided | untested | **missed** (independence axis) |
| 03 | 2026-08-02 | differently-biased-checkers (2026-08-01) | Qwen 3.6 35B-A3B 8bit | **VOID** | **VOID** | n/a | **still untested** | n/a |
**Trial 03 is VOID and is entered as VOID rather than omitted** — an absent row would read as a trial not attempted. Write-up: `fool-trial-03-2026-08-02.md`. Two independent failures, both found by reading the output, neither by a check:
1. **The harness certified a run with no answer.** Qwen emitted an untagged scratchpad (`"Here's a thinking process:"`, zero `<think>` tags), so the tag regex reported `reasoning_present: false` and recorded all 2,944 words of deliberation as the *answer*; the token ceiling then cut it off mid-sentence before the answer began. `degraded: null`. The guard tested the string for emptiness while its field claimed the result was sound. Trial 02's write-up had listed the inline-scratchpad problem as **Open**; the harness closed it on the assumption that inline meant `<think>`-tagged.
2. **The design forbade the region it was measuring.** The self-exemption axis lives in Part VII; the anti-echo constraint — added in trial 02 — instructs the reader to skip author-named limitations, and the scratchpad shows the model reaching Part VII and leaving it, *citing that constraint*. Silence on self-reference is therefore indistinguishable from obedience. **The axis was unmeasurable by construction**, independent of the truncation.
Trial 02's fix and trial 03's document were each sound alone; their interaction was not.
## What the log shows so far
@@ -47,6 +55,8 @@ The v1 Chamber (June–July 2025) ran written work past **two frontier models of
**No false-positive control has ever been run.** Every trial to date used a document with real weaknesses. The claim that the model will say *"nothing found"* on a sound document is **untested** — trial 02's apparent restraint was an artifact of a disabled reasoning mode. Until a clean document is run, the finding-rate cannot be distinguished from a production-rate.
*Still true after trial 03, which was attempted and voided (2026-08-02). Two attempts have now failed for two unrelated reasons, which is itself the finding: the obstacle is not the model but the instrument and the design around it.*
## Grading caveat, standing
Every grade above was assigned by the executor, whose own errors are among those being graded, and whose reading of what counts as "real" is the reading under test. The findings are individually checkable; the *grades* are not independent.