Repository path: workshop/experiments/E-20260828-forced-half/critic-response.md · rendered 2026-09-09

Page metadata (front matter)

type	`note`
id	`E-20260828-critic-response`
status	`frozen`
created	`2026-08-28`
updated	`2026-08-28`
links	`workshop/experiments/E-20260828-forced-half/design.md`, `workshop/experiments/E-20260828-forced-half/raw/critic.json`, `wiki/arms/ARM-radif.md`, `wiki/method-notes.md`

The pre-run critic, and what it changed — 16 findings, 14 accepted, 2 remedies substituted

Two seats on design v1 and one complete window in both arms, 2026-08-28, before any data call. P1 openai/gpt-5.6-terra (2,533 tokens, stop) and P2 google/gemini-3.6-flash (5,929 tokens, stop), cap 12000, neither truncated — note (brt) did not fire this time, and the cap was raised from 9000 on its record. $0.071679500. Both returned NEEDS-REDESIGN: P1 10 findings (4 BLOCKING), P2 6 findings (2 BLOCKING). Raw replies: raw/critic.json. Design v1 is in git history at 9a31299b; what runs is v2.

The five BLOCKING findings, and what each did

#	seat	finding	disposition
B1	`P1`	*A within-cell contrast is not position-proof against a disclosure × order interaction.* A disclosure that changes attention to line-ends can amplify or reverse first-position bias; because `REP` and `CHI` occupy different positions under the two orders, that shows up as a spurious d of opposite sign in the two orders, and pooling hides it.	ACCEPTED, and it changed the primary's unit. The inferential unit is now the window mean over both orders and all three seats, in which a symmetric position shift cancels exactly; and the order-split figures are reported for every primary, with a conclusion withheld where the two orders disagree in sign.
B2	`P1`	The two arms are whole independent renderings, differing in diction, syntax, rhythm and idiom, so a movement cannot be attributed to repetition versus chime.	FINDING ACCEPTED; REMEDY SUBSTITUTED. Its remedy — several translators, counterbalanced, more versions — is out of this project's reach. Instead the estimand is narrowed and named: what is estimated is the movement in preference between two fixed whole renderings caused by naming or supplying the source's line-end shape, never an isolated device effect. §5.0 says so and the result page will repeat it. This is the same repair `RS-20260826b` §6 made under the same finding.
B3	`P1`, and `P2`'s MAJOR 3	The conditions are not one-thing-at-a-time. `IS` is far longer than the others and supplies the source text; length, salience and cognitive load could move a choice by themselves.	ACCEPTED, and it bought a fifth condition. New `IL` — the stem plus a different Sa'di ghazal (۴۹, not in this experiment's material) in script and transliteration, at the same number of bayts as the window, introduced as "a different poem by the same poet, for comparison." It is length-, script- and form-matched to `IS` and is not the source of the passages. `P1b` is now claimed only against `IL`, not against `I0`. +54 calls.
B4	`P1`	The cell-level sign test treats 54 cells as independent when three poems and nine windows supply all the text variation; deterministic repeat calls are not independent readers.	ACCEPTED. The cell-level test is demoted to descriptive. The registered primary is at the window level, n = 9.
B5	`P2`	`IS` does not isolate a visual shape: these seats read Persian, so `IS` supplies the whole source, semantics included.	FINDING ACCEPTED; REMEDY SUBSTITUTED. The remedy — replace the Persian with non-linguistic glyphs — answers a different question. §2 is rewritten instead: `IS` supplies the source, and the contrast with `RS-20260827b` is no longer audible versus visible but a rhyme versus an identically repeated word, both of which a reader of the script can recover. The visual-shape framing of `v1` is withdrawn.

The MAJOR and MINOR findings

#	seat	finding	disposition
M1	`P1` 4, `P2` 5	The registered direction is a story, not a prior. `P2` supplies the counter-story with the texts in front of it: told about the form, a seat may penalise `REP` for its unrhymed repetitions rather than reward it.	ACCEPTED. `P1a` and `P1b` are now two-sided. Whichever way the movement runs is the finding, and a movement toward `CHI` is written into §7 as a substantive outcome rather than a falsification.
M2	`P2` 1	Power. At n = 9 a one-sided exact sign test needs 8 of 9; if one ghazal's three windows disagree the maximum reachable is 6 of 9. The registered primary could be incapable of passing.	ACCEPTED, and it changed the test. The primary is an exhaustive sign-flip cluster randomisation test on the nine window means — all 2⁹ = 512 sign assignments enumerated, statistic the mean of window means, two-sided. It uses magnitude as well as direction, so a consistent moderate effect with one dissenting window is reachable where the sign test would give 0.18. The exact sign test is reported beside it. `P2`'s own remedy (a mixed model) is declined: stdlib only, and an exhaustive permutation over 9 clusters is exact and assumption-free.
M3	`P1` 6, `P2` 6	`P2`'s specificity was post-selection (choose whichever condition moved, then compare with `IP`), and its criterion (iii) algebraically reduces to a direct paired contrast, making the `I0` term redundant — note (bsa) in its second firing.	ACCEPTED. Two contrasts are predefined and neither is selected after seeing anything: `I2` vs `IP` and `IS` vs `IL`, each a direct within-cell paired contrast. Holm across the confirmatory set of four.
M4	`P1` 7	The `C1` gate lets one model seat change the confirmatory set after registration, in either direction.	ACCEPTED, with a different fix. A `DIFFERENT` verdict on a real pair no longer removes a window. Every primary is reported twice — with and without the flagged windows — and a conclusion is claimed only where the two agree. The gate's information is kept and the degree of freedom is closed.
M5	`P1` 9	"Falsified by no shift" is wrong: a non-significant result at this power is not evidence of absence.	ACCEPTED. §7 now says unestablished, and the arm-closing sentence says what this project cannot establish, not what is not there.
M6	`P1` 10	The main run cannot separate policy from fixed-rendering quality; run a small factorial pilot first.	OVERRULED, with the reason. The fixed quality difference between the two arms is constant across the conditions being contrasted and cancels in every registered primary; what does not cancel is a disclosure × quality interaction, which is what B2's narrowed estimand declares rather than hides. A pilot on the same three pairs would answer nothing the main run does not, at the cost of a second dispatch.
M7	`P2` 4	Position bias varies with prompt length, so holding order fixed within a cell does not remove it: `IS`'s longer prompt could shift it.	ACCEPTED — this is B1 from the other side and takes B1's remedy. Averaging over the two balanced orders cancels a symmetric position shift exactly, which is why the window mean is now the unit; the order-split figures are the diagnostic.

What the critic did not change

The three ghazals, the six renderings, the nine windows, the four original conditions and the seats are all as v1 had them. Every amendment is to the analysis, the control conditions, or the wording of what may be concluded — nothing was rewritten to make a prediction easier to satisfy, and the one prediction whose direction was in doubt was made two-sided rather than re-argued.