Repository path: workshop/experiments/E-20260730e-rule-coverage/amendments.md · rendered 2026-09-09

Amendments A1–A6 — made on the pre-run critic's findings, before any rater call

Critic: moonshotai/kimi-k3 (P4), one call, $0.05598, provider Fireworks, in 4,540 / out 2,824, stop. Verdict NEEDS-AMENDMENT — six findings: two BLOCKING, two MANDATORY, two ADVISORY. All six accepted. One (Q5) is not amended but WITHDRAWN, which is more than the critic asked for and is explained below. Note (rr), twenty-second consecutive session in which an independent pre-run critic changed a design before money was spent.

Full text: runs/critic.txt. The C1 prompt and item pool are unchanged in every byte that a rater sees; only C2's labels, and the registered statistics, move.

A1 — Q7 measured convergence and called it the rule's silence (Finding 1, BLOCKING)

The critic is right and the defect is exactly the one it names. Q7 counted convergence and registered a prediction of non-convergence as evidence that F10 does not decide. But unanimous FLATTEN at 4 of 4 sites is the strongest possible evidence that F10 does decide — and the frozen Q7 would have scored it as the prediction failing, i.e. as evidence in the other direction. The statistic could not mean what the design said it meant.

Q7 is replaced by Q7′:

Q7′ (registered). Raters converge on a substantive verdict — all three KEEP, or all three FLATTEN — at ≤ 2 of 4 sites. Fails if substantive convergence occurs at ≥ 3 of 4.

And the direction is registered separately, because it decides what has to be corrected: unanimous FLATTEN at ≥ 3 of 4 sites REFUTES IR1. It would mean the literal reading of F10 is the right one, that the translator's ruling was a convenience, and that T-postmaster-R07-v1 and T-petits-poemes-R07-v1 both need an erratum saying the regime was applied against its own rule. That is the outcome most costly to the lead and it is named here, in advance, as a live one.

A2 — the C2 label set manufactured its own non-convergence (Finding 2, BLOCKING)

KEEP was defined as "requires, or at least permits and does not weigh against", which is the same state of the rule as UNDECIDED described twice. A rater concluding F10 is silent could defensibly return either, so disagreement between them was guaranteed noise feeding the Q7 count. The prompt is rebuilt with three mutually exclusive, jointly exhaustive labels:

• KEEP — F10's text weighs in favour of carrying the figure over.
• FLATTEN — F10's text requires the plain rendering.
• UNDECIDED — F10's text neither favours nor forbids; it does not settle the question.

runs/prompt-C2.txt is regenerated. No C2 call had been dispatched.

A3 + A4 — Q5 is WITHDRAWN, and finding it unrunnable cost the design its only defence against the gloss leak (Findings 3 and 4, MANDATORY)

The critic said Q5's gloss control was confounded with language and that its 0.15 threshold had no sampling justification. Both are true, and checking the first uncovered something worse.

The glossed flag was computed as "the site cell carries a gloss OR the text is the Bengali one". That second clause made every Bengali item glossed by fiat and manufactured the confound the critic detected. Computing the flag from the cell text alone — build_items.py, amended, with the sample and option order asserted byte-identical afterwards — gives the true counts:

Two glossed items in thirty-seven. No statistic can be computed on that, with or without a permutation null, so Q5 is withdrawn rather than downgraded. The honest consequence is that this run has no control at all on the lead-gloss leak, and it is recorded as owed rather than dressed as measured.

Reported in its place, predicted by nothing: lead-agreement and pairwise agreement broken out by source language. That contrast is confounded four ways — language, the log's code distribution, whether the translator glossed the site, and which of three texts it came from — and is reported as a description, never as a test.

And the structural point the critic's finding surfaces, which belongs in the result: a gloss exists exactly where the translator judged the source needed one, so gloss and source distance cannot be separated by sampling from logs that already exist. Separating them needs items built for the purpose.

A5 — the α fallback is pre-registered (Finding 5, ADVISORY)

If any rater omits a label:

• both the raw α (over the three-label scheme as designed) and the α over the reduced label set are reported;
• the raw α is the one compared to the 0.40 threshold, since it is the statistic over the scheme that was frozen;
• and a raw α that clears 0.40 while a label is unused is flagged and NOT read as a pass — this is the S056 defect (κ 0.809 → 0.066 once the third label became reachable, note (bdq)), and the flag is what keeps this run from repeating it.

A6 — the repeat control acquires a consequence (Finding 6, ADVISORY)

Q8′ (registered). If P1's self-agreement across the byte-identical C1 repeat is below 0.80 over the 37 items, every C1 figure is reported as instrument-limited and Q2 is not read as a measurement of the rule set — the same void condition F1 carries.

Reference: S063 measured 7.2% flips on a byte-identical same-day repeat of a presence task, i.e. self-agreement ≈ 0.93. A floor of 0.80 is below that and above chance (0.33).

Cost after amendment

Unchanged: C2's prompt is 3,150 bytes against 3,086, and no call count moves. Declared worst case stays $1.16, against today's headroom of $3.4762 less the critic's $0.05598 already spent.