Repository path: workshop/experiments/E-20260806c-source-grammar/critic.md · rendered 2026-09-09
Page metadata (front matter)
| type | note |
|---|---|
| id | critic-20260806c |
| status | frozen |
| created | 2026-08-06 |
| updated | 2026-08-06 |
| links | workshop/experiments/E-20260806c-source-grammar/design.md, workshop/experiments/E-20260806c-source-grammar/runs/critic-response.md, config/models.md |
Pre-run critic pass — E-20260806c, and what was accepted
Seat x-ai/grok-4.5 (P3), one call, max_tokens 24,000, temperature 0.
Cost $0.0418704 (per-response usage.cost); 7,031 prompt / 4,671 completion tokens, of which
2,898 reasoning; finish_reason: stop. Raw body: runs/critic.raw.json; response verbatim:
runs/critic-response.md; prompt: runs/prompt-critic.txt.
Verdict: NEEDS-AMENDMENT. Ten findings, six BLOCKING. All ten accepted. Every amendment below
was applied to design.md §8 before measure.py was written and before any statistic in this run
existed.
The two that changed most: F1 removed the arm-closing power of the predictions the translation limb generated, and F3 added a gate that the predecessor run's own stored numbers suggest will fire. Neither was in the design when it was frozen.
| # | blocking | finding, in one line | disposition |
|---|---|---|---|
| F1 | yes | Two translator-coded censuses on two unmatched loci cannot ground corpus-level grammar predictions; seven confounds named, including that the coder is the translator | accepted → A1 |
| F2 | yes | The closure rule is a disjunction and lets the arm close on a null the data do not support | accepted → A2 |
| F3 | yes | FC4 misses the case where the share component is small but the whole effect lives inside speech; 80% is arbitrary; two original arms have almost no speech at all |
accepted in all three parts → A3 |
| F4 | yes | FC1 shows the machinery sees a large contrast and says nothing about power against the effect P4 claims to have ruled out |
accepted → A4 |
| F5 | yes | Holding the hand fixed does not hold genre fixed; P1 could pass on genre baseline mismatch alone |
accepted → A5 |
| F6 | yes | An unweighted pooled mean gives equal vote to low-precision cells; P1 can pass or fail on cell-size geometry |
accepted → A6 |
| F7 | no | §2.4's "two ways of computing the same diagnostic" is policy, not epistemology: the predecessor's published sentence is false under the analysis-consistent method | accepted → A7 |
| F8 | no | your sits in both scored classes, so P1 and P2b are not separable |
accepted → A8 |
| F9 | no | P2a's cell set is two against two and heterogeneous |
accepted as a declared limit; subsumed by A1 |
| F10 | no | Delta-z computed once outside the permutation loop can inflate Type I for the profile statistic | accepted as a declared limit; subsumed by A4 |
The one place the critic and the design still differ, declared rather than silently resolved
F2's remedy says that if P1 fails, FC4 does not fire and P3 is null, the run must write
"unresolved — replication failed, mechanism unadjudicated" and not close the arm. A2 adopts
that verdict for the scientific claim and separates it from the anchor decision, which is the
arm's actual Done when: an unreplicated displacement whose mechanism is unadjudicated is not
something a naturalness anchor can be built on either, so the shelf decision is answerable while
the mechanism question is not. The arm may therefore close on the anchor question while the run
reports the mechanism question open. That distinction is A2's and is stated on the arm page as
well, so a later session can disagree with it in one place.
What the critic did not question, and should have been able to
It accepted §2.5's declaration of prior knowledge without testing whether the declaration is complete. It cannot: the declaration is the lead's own account of what the lead had looked at. That is a standing hole in every pre-registration this project writes, and it is not repaired by a critic pass.