David F Glidden
cd2edaa3c9
Third and largest finding from the trial-03 post-mortem. The pulling thread — in MEMORY.md and in the previous wrap — named trial 03 'the Fool's false-positive control'. Trial 03's own pre-registration says it asks whether the checker shares the 2025 archive's self-exemption disposition, and its grading section states that 'the false-positive rate is still unmeasured'. The pre-registration knew what it was. A false-positive control needs a SOUND document, so that 'nothing found' is the correct answer. Trial 03's input was chosen with five pre-registered weaknesses, deliberately, because absence of the strong hit is only interpretable if performance is otherwise competent. The ground-truth list exists to establish that the document is NOT sound. They are different experiments. The wrap held the contradiction in one paragraph — calling trial 03 the control while saying the control requires a sound document trial 03 does not use. It survived the wake, was restored as the thread, and was 'substrate-checked': the check verified the M4 was up and that trial 03 had not run, and never asked whether the trial was the thing the thread said it was. Checking that a claim's referent exists is not checking that the claim is true. The conflation then reached the run record's note field, which is preserved with the error in it. Consequence, larger than trial 03: the false-positive control has not merely gone unrun, it has never been DESIGNED. It needs a document believed sound, and soundness cannot be known by construction. That choice is a fork, and it is surfaced rather than taken.
65 lines
8.3 KiB
Markdown
65 lines
8.3 KiB
Markdown
# Fool trial log — the running record
|
||
|
||
*The correlation data the differently-biased-checkers doctrine says is owed and has never been produced. Built at n=2 rather than when it becomes a problem — the lesson of the skill-harvest register, which grew to 166 KB before anyone noticed it had stopped being readable. One row per trial. Detail in the per-trial files.*
|
||
|
||
**Doctrine under test:** `differently-biased-checkers-JURIST-PACKAGE-2026-08-01.md` (ESCALATE, unruled). Its claim: oversight needs checkers whose contaminations do not point the same way. Its named falsifier: *if the parties' misses correlate — if what one misses the others reliably miss too — the principle is false for that configuration.*
|
||
|
||
## Standing protocol
|
||
|
||
1. **Only documents with known ground truth** — packages the jurist has already ruled — so hits and misses are countable.
|
||
2. **Withhold the ruling**, the addendum, and any hint of what was found weak.
|
||
3. **Pre-register the grading before the run.** Written down, not remembered.
|
||
4. **Prompt gives form, not target.** No steer toward any part of the document; explicit anti-contrarian and anti-echo clauses.
|
||
5. **`enable_thinking` ON.** Established load-bearing in trial 02 — off produces silence, not brevity.
|
||
6. **One variable per trial.** Violated in trial 02's first run; the result was uninterpretable and had to be re-run.
|
||
7. **No standing granted to the Fool.** Its findings earn a hearing by being checkable, never by role (steward correction, 2026-08-02).
|
||
8. **The protocol is a file, or it is not a protocol.** Every trial runs through `fool/run_trial.py`, with the prompt as a versioned file hashed into the run record, and every sampling parameter recorded including defaults. Established 2026-08-02 after discovering trials 01–02 are **not reproducible** — no script, no verbatim prompt, no temperature, top_p, max_tokens or seed. The same failure destroyed the v1 Chamber's GPT-side protocol, which lived as configuration inside a hosted product; the Claude-side prompt from the same day survives because it was a file.
|
||
9. **An empty answer is not a finding of silence.** If the model emits only a reasoning trace, or nothing, the harness marks the run `degraded` and the result may not be graded as restraint. This is trial 02's error made structurally impossible.
|
||
|
||
## Trials
|
||
|
||
| # | Date | Document | Model | Strong | Moderate | Echo | False-pos | Jurist's central catch |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| 01 | 2026-08-01 | PENDING-88 skill-harvest FIX lane | Qwen 3.6 35B-A3B 8bit | MISS | **MET ×2** | n/a | untested | **missed** (narrower test is less safe) |
|
||
| 02 | 2026-08-02 | order-attestation (2026-07-29) | Qwen 3.6 35B-A3B 8bit | MISS | **MET ×2** | avoided | untested | **missed** (independence axis) |
|
||
| 03 | 2026-08-02 | differently-biased-checkers (2026-08-01) | Qwen 3.6 35B-A3B 8bit | **VOID** | **VOID** | n/a | **still untested** | n/a |
|
||
|
||
**Trial 03 is VOID and is entered as VOID rather than omitted** — an absent row would read as a trial not attempted. Write-up: `fool-trial-03-2026-08-02.md`. Two independent failures, both found by reading the output, neither by a check:
|
||
|
||
1. **The harness certified a run with no answer.** Qwen emitted an untagged scratchpad (`"Here's a thinking process:"`, zero `<think>` tags), so the tag regex reported `reasoning_present: false` and recorded all 2,944 words of deliberation as the *answer*; the token ceiling then cut it off mid-sentence before the answer began. `degraded: null`. The guard tested the string for emptiness while its field claimed the result was sound. Trial 02's write-up had listed the inline-scratchpad problem as **Open**; the harness closed it on the assumption that inline meant `<think>`-tagged.
|
||
2. **The design forbade the region it was measuring.** The self-exemption axis lives in Part VII; the anti-echo constraint — added in trial 02 — instructs the reader to skip author-named limitations, and the scratchpad shows the model reaching Part VII and leaving it, *citing that constraint*. Silence on self-reference is therefore indistinguishable from obedience. **The axis was unmeasurable by construction**, independent of the truncation.
|
||
|
||
Trial 02's fix and trial 03's document were each sound alone; their interaction was not.
|
||
|
||
## What the log shows so far
|
||
|
||
**The correlation the doctrine warns about is present, and localised.** 2/2 trials missed the jurist's sharpest contribution. That is not a general blind spot — the Fool produced four real findings across the two trials — it is a *specific* one, and it now has a shape:
|
||
|
||
- **Jurist** — errors of **inference**: where an argument stopped counting, where an alternative is less safe than it appears.
|
||
- **Fool** — **unestablished premises**: scope-exhaustiveness, downstream mechanism drift, patterns assumed sound.
|
||
- **Executor** — substrate, containment, arithmetic; and reliably *not* its own inference errors.
|
||
|
||
**The open question this poses, and it is the sharpest available experiment:** is the inference-level miss a property of *Qwen*, or of *any non-jurist reader*? A second, differently-formed model run on the same two documents answers it. If it also misses, the gap is structural and no model choice closes it. If it catches, model choice matters far more than assumed.
|
||
|
||
## The 2025 arm — a prior measurement, found not run (2026-08-02)
|
||
|
||
The v1 Chamber (June–July 2025) ran written work past **two frontier models of the moment**, preserving both raw outputs unmerged. Read at the steward's direction 2026-08-02; analysed in `differently-biased-checkers-ADDENDUM-1-2026-08-02.md`.
|
||
|
||
**Mutual divergence in 3 of 3 pairs where the instruction was comparable** — each party landing precise textual hits the other missed entirely. The one non-mutual pair is the one whose prompt was most heavily compressed.
|
||
|
||
**Why it matters to this log specifically.** These trials run at a large **capability gap** — a ~35B local model against a frontier one — so divergence here has an alternative explanation: a weaker checker diverging by being *weaker* rather than by being *differently formed*. The doctrine is about different bias, not different capability. The 2025 parties were roughly matched, so their divergence **cannot** be a capability artifact. That is the arm these trials structurally cannot produce, and it returns the same result.
|
||
|
||
**And it supplied trial 03's hypothesis.** One 2025 checker exempted the venue it was performing inside, while attacking freely elsewhere, under a system-level instruction reading *"No softening."* The steward had recorded the same disposition in a user guide dated **2025-01-20**: *"May smooth over tensions"* — a bullet, no terminal period. Trial 03 tests whether ours shares it.
|
||
|
||
## Untested, and load-bearing
|
||
|
||
**No false-positive control has ever been run.** Every trial to date used a document with real weaknesses. The claim that the model will say *"nothing found"* on a sound document is **untested** — trial 02's apparent restraint was an artifact of a disabled reasoning mode. Until a clean document is run, the finding-rate cannot be distinguished from a production-rate.
|
||
|
||
*Still true after trial 03 — and more sharply than "unrun" conveys. **The false-positive control has never been designed, let alone run.** Trial 03 was carried in memory and in the session wrap as "the Fool's false-positive control", but its own pre-registration says it asks whether the checker shares the archive's self-exemption disposition, and its grading section states plainly that "the false-positive rate is still unmeasured". A false-positive control needs a **sound** document so that "nothing found" is the correct answer; trial 03's input was chosen with **five** pre-registered weaknesses precisely so that competence could be verified. The two are different experiments and were conflated in the record.*
|
||
|
||
*Designing it requires a decision not yet taken: what stands in for a sound document, given that soundness cannot be known? The candidates are a document that has already survived jurist review and steward placement, or a constructed one whose weaknesses are known to be absent. Until that is settled, no trial run can close this item.*
|
||
|
||
## Grading caveat, standing
|
||
|
||
Every grade above was assigned by the executor, whose own errors are among those being graded, and whose reading of what counts as "real" is the reading under test. The findings are individually checkable; the *grades* are not independent.
|