Repository path: framework/control-arm-spec.md · rendered 2026-09-09

The control-arm specification — how a paired comparison must be built and analysed

What this is. ARM-framework step 3, and the deliverable RS-20260727c-arm-identifiability §11 item 1 asked for. That page prescribed a repair in two forms and did not choose between them. S046 priced both and neither survives as prescribed (RS-20260728c-length-matching). What is below is what survives, plus the two things the pricing found that neither form anticipated.

Status. Adopted by S046 on measurement. It is not a ratified decision — no independent vote has been routed through it — and a later session may put it to the ratification protocol. It binds designs by being cited, not by authority.

Standing gate, added 2026-07-30 (S061), and it is why this paragraph is no longer only a status note. "Ratify framework/control-arm-spec.md" sat in wiki/backlog.md from S051 and reached the review-or-retire rule's ten sessions without being done. Its declared trigger — "the next design that builds a paired comparison" (framework/closure.md §4) — fired at E-20260729c and was not routed, which config/models.md already records, and nothing since has routed it either. The row is therefore retired from the backlog and converted into a gate here, on the S060 finding that a signal a session is free to decline is a signal that will be declined while something newer is available:

The next design that cites any rule on this page as binding must route the ratification vote in the same session, before its own dispatch, and record the outcome here. A design that cannot afford the vote must say in its own text that it is citing an unratified spec and why that is acceptable for its claim. Either is an honest ending; citing this page silently is not.

This is a gate, not a unit: it belongs to the session it blocks and is timeboxed to it (one routed vote, ≈$0.05), exactly as wiki/tracks.md §T6 prescribes for instrument work. Converting the obligation into a condition at the point of use, rather than a row that ages, is the whole point of the change.

THE GATE FIRED AND WAS ROUTED — 2026-08-01 (S083), E-20260801f-tierD-run §13. Seat P3 x-ai/grok-4.5 (a panel member, so a non-Anthropic panel vote per charter §8; deliberately not one of that design's jurors P1/P2/P5, so the voter is not ratifying a spec it is about to be measured under). Provider xAI, stop, 73.1 s, $0.0240956. Record: workshop/experiments/E-20260801f-tierD-run/ratify/.

Verdict: RATIFY-WITH-AMENDMENT. Amendment A1 is applied below, verbatim as voted. This page is now ratified as amended, and the S051 backlog row that became this gate is discharged.

And the vote overturned the citing design's own reading, which is why routing it was worth $0.02. That design stated that R1–R4 did not bind it because it reports no stratified estimate. The vote returned INCORRECT: "R1 is a pooling and reporting rule for every paired comparison in scope … not a rule that switches off when the outputs are threshold counts rather than CI-backed preference proportions." The design was amended before it dispatched anything.

The vote's own strongest counter-argument, recorded because it asked for one and it is not weak: this page is provisional: true, inherits an uncalibrated jury, and hangs R4's sample-size targets on post-hoc width calculations from one small study, so a cleaner vote would have been REJECT until the rules are re-measured on a calibrated jury, or RATIFY of R1, R5 and F1 only.

Scope. Any design in which a jury compares two texts that are the same content under two conditions: regime comparisons, Tier D sham and targeted arms, held-out arms, anything paired.

The five rules

R1 — Never pool a paired comparison across length-sign strata. Report per stratum, always.

This is the one half of RS-20260727c §11 item 1 that the pricing left intact, and its evidence is unchanged: on S010's TEMP arm the pooled preference sits near 0.5 while the two length-sign halves sit at roughly +0.15 and −0.15 (RS-20260727c §6). A pooled figure near chance is not evidence that the manipulation did little; it can be the exact average of two opposite-signed halves. RS-20260724-selfrevise-first's inference from its own control is the case in point and remains withdrawn.

Reporting form: per-stratum mean, per-stratum item count, per-stratum interval. A pooled number may be shown only alongside both strata and never as the headline.

Applicability (amendment A1, ratification vote 2026-08-01). R1 binds every paired comparison in scope, including designs whose primary outputs are threshold counts, pass/fail tallies, or mean score drops against a pre-registered criterion, not only designs that report preference proportions with intervals. The required minimum is: split every headline aggregate by length-sign stratum (per-stratum item count and per-stratum raw aggregate). Intervals follow R3 when the design reports them; they are not required in order for R1 to apply.

R2 — Do not report a stratified figure as an estimate.

At the item counts this project has run to date (n ≤ 10 per stratum), a stratified figure is not an actionable estimate (amendment A1). That is a fact about achieved n and interval width, not a permanent property of stratification. When a future design meets R4's item counts, R2 ceases to forbid estimate language for that design.

Measured (RS-20260728c §3). A cluster bootstrap over items, 95% percentile, against a criterion of interval width ≤ 0.30:

Stratification is a reporting discipline, not an estimator. It stops a specific false inference (R1). It does not deliver a number anyone may act on at n ≤ 10.

R3 — An interval-width criterion is not a validity criterion, and it inverts at small n. Use a hard minimum item count.

The MAIN Δ-positive stratum has two items and passes the width criterion on 3 of 5 senses, with intervals narrower than the six-item strata (0.250 against 0.361). This is not precision. A bootstrap over two numbers can only ever resample those two numbers, so it reports the spread of a two-point set as if it were the spread of a population. The tell is that the anticonservative binomial interval — the one that wrongly assumes 24 independent votes — is wider there (0.523 against 0.250): when the interval that ignores clustering is wider than the one that models it, the clustered one has run out of clusters.

This defect was in a criterion this session registered in advance, and it was found because the design registered A4 — run the same computation on the degenerate case — as a check on itself. Note (p): an instrument needs a case it should pass and a case it should reject, and A4 was the second.

The rule: a stratum with fewer than 5 items yields no interval at all, only its item count and its raw per-item values. Below 5, report the values and no summary.

R4 — The minimum n is a computed number, and it is four to five times what this project has been running.

Post-hoc from the item-level standard deviations (RS-20260728c §3, labelled post hoc there and here), the n at which a 95% interval would reach width 0.30:

So a paired comparison needs on the order of 22–30 items per stratum — roughly 45–60 pairs — to say anything per stratum. S010 ran 12 pairs total, 6 per stratum. The shortfall is a factor of four to five, and it is an affordability finding that belongs in a design rather than in a later post-mortem (note (bci)).

A design that cannot afford that many pairs must say so and must not report a stratified estimate. It may still report R1's split as a diagnostic — the split is what detects the confound — while claiming no estimate from either half.

Disclosure and diagnostics when n is short (amendment A1). A design that does not claim stratified estimates still (i) states that it lacks R4's per-stratum n if it does, and (ii) remains under R1: it may use the length-sign split only as a diagnostic and must not headline a pooled aggregate as evidence that a manipulation (or the jury's detection of one) was null, small, or decisive. Pre-registered threshold rules do not exempt the design from (i) or (ii).

R5 — Record Metric A and a correlation at freeze time, for every paired design.

tools/metric_a.py (moved into tools/ at S046 as the gate on this step). One call. Identifiability and cue-use are independent properties and each instrument sees one of them — note (bck), established on the one dataset that contains both cells.

At n = 1 metric_a returns A: None and the sign, and that is the correct output rather than a failure: A at n = 1 is 1.000 or 0.000 by construction.

What the pricing found that neither prescribed form anticipated

F1 — Length-matching is achievable, and it makes a THIRD text rather than a matched arm.

Measured on real prose at n = 1 (RS-20260728c §4): a 421-word revision was brought to its draft's 407 exactly, at 6 edit sites, 18 tokens touched to move 14 — an excess over the arithmetic floor of 4 tokens.

The classification of those six sites is the finding:

Against a governing coincidence floor of 0.0665 (the rate at which an anchored run spanning a change recurs in the draft by chance), so REVERT = 0 is a real zero and not a masked signal.

Consequence, and it is the operational one. A length-matched arm is not the treatment arm and not the baseline arm. It is a third condition. A design that matches must declare the matched text as its own condition and must not call it "the control arm, matched" — that name asserts an identity the measurement does not support.

And the honest bound: n = 1. One passage, one language pair, one translator, one matching pass, in a limb the design labels a non-generalizable case demonstration after the critic required it. This establishes that matching can produce a third text; it estimates nothing about how often, and the registered prediction that matching would revert the arm failed.

F2 — The jury's length response is graded, not flat — which is an argument FOR matching, against this session's own expectation.

RS-20260728c §2. On TEMP the association between |Δwords| and preference-for-the-longer-text is positive and material (pooled ρ = +0.470, descriptive only), and on two of five senses it survives a permutation test over 12 items: voice p = 0.011, literary-quality p = 0.002. The smallest gap in the set — one word — sits at preference 0.500 exactly; the largest — 151 words — at 0.700.

The lead had registered sign-only (P1) and that a specification adopting matching would count against its expectation (P5). Both went the other way. A graded response is precisely the condition under which matching to a tolerance would work, so the evidence points at the option this session expected to reject.

Three qualifications, and they are why R1–R4 are not rewritten around it. 1. Association, not mechanism. The critic's G1 was accepted before the numbers existed: item quality can correlate with |Δ| under a sign-only mechanism, and noise can erase correlation under a graded one. A positive ρ does not establish a dose response. 2. The design could not have seen three of the five senses. The detection floor at n = 12 is |ρ| ≈ 0.58; accuracy (0.483) and naturalness (0.280) are below it. Two significant senses out of five is two out of the two the instrument could reach. 3. Nothing here is a jury verdict. Tier D is NOT PASSED. These are facts about a design, and a graded response measured on an uncalibrated jury does not license a tolerance rule.

So the specification does not yet permit a tolerance, and names what would settle it: the same association measured on a comparison with R4's item counts, on a calibrated jury. Until then, matching is permitted under F1's declaration rule and is not required.

F3 — Stratifying on sign controls the sign confound and nothing else, and F2 makes that bite.

The critic's G9, accepted before the numbers: A3 measures precision, not validity. Stratifying on the sign of Δ removes the sign confound within a stratum by construction; it does not remove magnitude confounding inside it. F2 turns that from a hypothetical into a live one — if preference tracks |Δ| and a stratum spans 1 to 151 words, the stratum is confounded internally.

The rule that follows: every stratified report carries a within-stratum magnitude check — the same association statistic, computed inside each stratum — or states that it was not computed. Designs that report no stratified estimate still run the within-stratum magnitude check when they report R1's diagnostic split, or state that it was not computed (amendment A1).

What this specification does NOT do

• It does not repair the S010 regime comparison. Both prescribed repairs have now been priced and neither delivers what RS-20260724-selfrevise-first wanted from its control. A re-run needs more items, not a better analysis (R4). Candidate #12 of the traceability inventory stays inadmissible, and its second defect stays undischarged.
• It does not license any recommendation to a translator. Nothing here is about how to translate. It is about how to build a comparison, and it enters the inventory as a procedural candidate about the project's own method, not as framework content.
• It is not evidenced on a calibrated jury, and every number in it inherits that (provisional: true).