Repository path: workshop/experiments/E-20260728g-scale-usage/critic/dispositions.md · rendered 2026-09-09
Page metadata (front matter)
| type | note |
|---|---|
| id | E-20260728g-critic-dispositions |
| status | frozen |
| created | 2026-07-28 |
| updated | 2026-07-28 |
| links | workshop/experiments/E-20260728g-scale-usage/design.md |
Pre-run critic pass — findings and dispositions
Critic: x-ai/grok-4.5 (panel role P3), provider xAI, 35s, in 5,411 / out 1,532, finish_reason: stop, temperature 0.2, reasoning: {"effort":"low"}. $0.0197964, 34% of the $0.058 worst case. First call, no fall-through needed. Raw body: ../runs/critic__grok-4.5.json.
Verdict: NEEDS-REDESIGN. Nine findings. All nine accepted, five of them by withdrawing or narrowing a claim the design had made. The design is amended to v2 before any statistic is computed; design.md §11 carries the amendment record and the section text is amended in place.
Why P3 and not P1. P1, P2 and P5 are the three jurors whose stored scores are the entire dataset. A subject cannot critique the analysis of itself. P4 is off the call list (note (b)). That left P3, which is also the panel member this project has used least.
| # | finding | disposition |
|---|---|---|
| A1 | The §8 concession that "material heterogeneity" and "presence of damage" are the same thing in this data destroys P1's licensed reading ("the gate is sited on the arm where the jurors move least") as a general siting claim. | ACCEPTED, claim narrowed. §6's P1 row is rewritten. What survives is not a claim about neutrality at all: the gate is read off the arm where these jurors' dispersion is lowest, and this data cannot say whether that is because the material is neutral or because it is undamaged — but under either mechanism a dispersion figure read there does not describe the juror's dispersion on the arms the gate licenses. That is weaker than what v1 wrote and is still the finding that matters for siting. |
| A2 | The best finding. P3's reading — a gate that would have stopped runs whose detection later fired "is not conservative; it is measuring the wrong thing" — is unsound. A gate is prior; detection is posterior. Being blocked-then-successful shows conservatism, not invalidity. | ACCEPTED, and the inference is re-routed. §5's P3 and §6's P3 row are rewritten. P3 now licenses an operating characteristic and nothing more: this gate, at this threshold, would have blocked N of 2 runs in which every juror separately went on to produce the targeted detection outcome — a false-negative count on the only two cases available, which is a cost, not a proof. The claim that the statistic measures the wrong quantity must now rest on P2 and P4, which are about the statistic itself, and not on P3. v1 had the load in the wrong place. |
| A3 | SD(L) pools 48 non-exchangeable numbers; sense means, item difficulty and slot offsets all inflate it relative to difference capacity, while integer bunching deflates it. | ACCEPTED, and it adds a statistic. §3 gains SD(L|sense) — the same dispersion with each sense's own mean removed within the cell — reported beside SD(L) everywhere. The gap between them is the amount of a juror's apparent "scale usage" that is really the juror marking cultural-mediation higher than accuracy. This is the per-juror analogue of exactly what RS-20260727b §4 did to the pooled statistic, and not doing it here was an inconsistency. |
| A4 | P2's conjunct SD(L) > 0.40 is inert — chosen so the prediction is nearly certain to hold. |
ACCEPTED, threshold replaced by a dissociation. P2 is restated to require that the juror with z(D) ≥ 0.50 is not the juror with the lowest SD(L) in that run. A magnitude that any spread clears is replaced by a rank contrast that can fail. |
| B | The Q4 replacement (a positive control on a constructed known-different pair) escapes D-20260725-06's prohibition only if the fault is fixed against an external damage specification rather than lead judgment; and passing it establishes sensitivity to that fault class at that dose on that passage, not a general capacity to express 0.75 across six senses. |
ACCEPTED in full, both halves. The distinction the critic requires is the one charter §5.2 already imposes: O4's catalogue is external — documented accuracy failures in published translations (A-shaw-spider-thread; Swann 1974 on Turney; A-chekhov-pari on Koteliansky & Murry 1915) — and lead-invented operators are forbidden. variant-F1.md is therefore built to O4's four documented failure types with the type named per site, and any site not instantiating one of them is not admissible. The limit is adopted verbatim as a standing caveat on the result page, and the critic's alternative — direct max\|d\| / mean\|D\| thresholds on the arms that matter — is recorded as a live competing option that the result page must weigh rather than ignore. |
| C1 | P2 is the weakest falsifiable prediction; the P3 row of §6 is overreach. | ACCEPTED — both handled under A2 and A4. |
| C2 | P4's claim and its falsification do not match. It is stated as a non-existence claim over all thresholds and falsified by a rank-ordering test that is necessary but not sufficient. | ACCEPTED, and P4 is split. P4a — every (run × juror) cell reaches the criterion margin on the targeted arm — is the separability question, stated so that holding it means no positive threshold is licensed at all. P4b — the rank ordering by sham SD(L) does not match the ordering by realised margin — is the monotonicity question. Each is separately computable and separately falsifiable, and §6 says what each combination licenses. |
| C3 | §6's table has no row for a split outcome (a prediction holding in one run and not the other), and no row for the UNEVALUABLE path §7.4 already provides. |
ACCEPTED; both rows added. |
| D1 | §7.2's fallback — a run whose slot convention cannot be re-derived "contributes to L-statistics only" — silently changes the composition of what is compared across runs. | ACCEPTED, fallback deleted. A run whose slot convention cannot be re-derived is excluded entirely, loudly, and the exclusion is the headline of the result. |
| D2 | A verifier sharing no code still misses a wrong slot convention, because both scripts read the same manifest the same wrong way. The §10.6 regression test only covers the S034 sham pool. | ACCEPTED, and it produces a genuinely independent check. If the convention were inverted, every d would flip sign. The targeted arms have a known ground truth — the reference is the undamaged text — so the verifier now asserts mean d > 0 on every targeted cell, which fails loudly on an inverted convention and does not depend on reading the manifest at all. |
| D3 | §4.5 says per-juror detection is computed "the same way as S034" without freezing the rule text, which invites silent drift. | ACCEPTED; the unit rule is now quoted verbatim in §3. |
| E1 | The §6 vocabulary prohibition bans calibrating / validating / passing but does not bind the validation-shaped sentences the design is actually reaching for. | ACCEPTED; the prohibition is rewritten to bind by form of inference rather than by word list. |
| E2 | Step 7 builds the fault variant on the same page that argues for the replacement it serves — control materials not frozen before the design that uses them. | ACCEPTED as a statement rather than as a change. The variant is materials and is used by no statistic on this page; it is frozen in its own commit, and any future design that uses it as a control must freeze that design after it. Written into §4.7 so the constraint travels with the artifact. |
One thing the critic got wrong, recorded because the pattern matters. It wrote that P2 "almost cannot fail as stacked" and separately that the budget is "fine… not mis-sized" — both correct — but it also asserted that §7.2's fallback breaks "cross-run comparability of 'per-juror gate'" without noticing that v1 never pooled across runs. The disposition above deletes the fallback anyway, on the stronger of the two reasons. Every structural claim the critic made about the design text was re-checked against the text before being accepted, which is the standing instrument caution from S043 and S049 (the critic there cited 27 id/position pairs and 12 were wrong).