Repository path: workshop/experiments/E-20260727-sham-decisions/design.md · rendered 2026-09-09

Frozen design — can a "quality-neutral" sham be built without overwriting the translator's decisions?

Frozen 2026-07-27 (S040), at a commit that contains this page and T-signal-R04-v1 and contains no sham substitution. The translation and its twenty-five-decision log were frozen one commit earlier (92bd3b6), before this page existed.

0. Why this exists, and what it is not

ARM-tierD closed with two named defects in the Tier D instrument, both of them arithmetic: §6.5's sham band has a branch whose null probability is 0.534, and §10's scale-usage gate is stated on the pool where it should be stated per juror. ARM-tierD-repair step 1 rebuilds both, and that work needs no experiment — it is enumeration.

This page asks the question underneath the arithmetic. §6.5 says the sham is "eight neutral substitutions" under four constraints — free variation only; word-, sentence- and paragraph-count neutral; no archaising; no sense shift — and concedes in the same breath that "a perfectly neutral sham is not constructible", because "any eight substitutions move [a considered text] off a local optimum". That concession has been carried, unexamined, through two Tier D runs. Nobody has ever built a sham against a text whose decisions were written down, so nobody has been able to say what a sham substitution actually is with respect to the translation it edits.

This is not a rescue of S034. The S034 verdict stands as pre-registered and is not reopened here (RS-20260726d-tierD-heldout, §1). Nothing on this page changes any figure that run reported.

This is not a jury run. No API call is made and no quality claim is asserted about T-signal-R04-v1 — the lead never judges its own translation (charter §5). Every quantity below is a token-overlap count computed by a committed script.

1. Question

Given a translation whose decisions are recorded, can eight sham sites satisfying §6.5's four constraints be placed so that they do not overwrite those decisions?

If they can, "free variation" names a real class of loci that sit outside the translator's choices, and the sham arm is measuring edit-presence as it claims. If they cannot, the sham is a degradation operator whose target sense is unnamed, and repairing its band's arithmetic does not repair it.

2. Materials

T-signal-R04-v1 — Garshin, «Сигнал» ¶2, 263 Russian words → 368 English words, translated in session under R04, frozen at 92bd3b6 with a translator's log carrying 25 decisions, each anchored to a verbatim, unique substring of the English. Uniqueness was asserted before the freeze; the analysis script re-asserts it and fails loud.

Scale parity with the Tier D items is deliberate: the run's references were 292–394 English words and carried 8 sham sites, so this passage takes 8 sites at the same density.

The comparator measurement is complete and clean (0 shared 7-grams against Seltzer 1917; longest common run 6 tokens, against a null-control floor of 4 over 88,012 words). It is recorded on the translation page. It matters here for one reason only: it is what licenses treating the twenty-five logged decisions as decisions rather than recollections.

3. The decided-token set

D = the union, over the 25 anchors, of the token indices they span in the translation. T = all tokens. Coverage c = |D| / |T|.

Tokenisation is defined in analysis/measure.py and is deliberately crude — split on whitespace, strip leading/trailing punctuation, keep case — because nothing here turns on tokeniser subtleties and a crude tokeniser is auditable at a glance. Anchors are located by exact substring match on the raw English and mapped to token spans; a non-unique or absent anchor is a fatal error, not a warning.

The header of the log names four items as explicitly not decided (versts, the Line, Kherson, the Don country). They are not in D. That exclusion was written into the frozen log before this page existed and is not revisited.

4. Procedure, in this order

1. This page is frozen. (Commit contains this page; contains no sham.)
2. Compute c from the frozen log, and record it. This is a property of the translation and involves no sham.
3. Construct the sham. Eight substitutions in T-signal-R04-v1, under §6.5's four constraints verbatim, chosen by the lead exactly as the operator is actually deployed — the lead picks both the site and the substitution. Each is logged in sham.md with its span, its replacement, and the constraint check.
4. Measure k = the number of the 8 substituted spans that intersect D, and j = the number whose replacement is a reading the log names as a live-but-rejected alternative.
5. Test k against the null of §5.
6. Verify: a second script recomputes c, k, j and the null distribution from the frozen log and sham.md alone.

Rule 3 carries one prohibition and it is the whole discipline of this page: the translator's log is not opened between the freeze of this design and the freeze of sham.md. The lead wrote that log an hour earlier and cannot unknow it; what it can do is not re-read it while choosing, and say so. See §7.

5. The null, and what fires

Null model. The 8 sham spans are placed independently of D: their observed span-length multiset is kept, and the spans are re-placed uniformly at random on the token line subject to non-overlap and to staying in bounds. 100,000 draws, seed 20260727, giving the null distribution of k.

The length multiset is taken from the observed sham because the alternative — assuming a length — would be a free parameter chosen after seeing the data. Keeping lengths fixed and permuting positions isolates exactly the quantity at issue: placement.

The firing rule, pre-registered.

Reachability, both directions (standing critic disposition 5). k ranges over 0–8 and both branches are attainable: a sham placed entirely inside D gives k = 8, a sham placed entirely outside gives k = 0. The rule can fire and can fail on the same materials.

Power is limited and the limit is stated now. With 8 sites the test can distinguish near-total avoidance from base-rate placement and nothing finer. A one-sided p < 0.05 requires k to be roughly 2–3 below the null mean, depending on c. A "FAILS" verdict is therefore the weak one: it is what the design returns whenever the lead does not strongly avoid D, including if the lead merely placed sites at random. That asymmetry is in the direction that flatters the sham premise's rejection, and it is why §6's prediction 3 is stated as a separate, harder claim.

6. Predictions, written before the sham exists

1. c ≥ 0.50. The log's anchors cover at least half the translation's tokens. (Basis: 25 anchors over 368 words, several of them clause-length.)
2. k ≥ 6 of 8. (Largely determined by prediction 1 — stated anyway, because a prediction that follows from another is still falsifiable and its failure would be informative.)
3. The premise FAILS: k ≥ the null mean. This is the substantive prediction and the one that could most easily be wrong. It says the lead, trying to edit neutrally, does not systematically seek out the loci it did not deliberate over — that "free variation" is not something a translator can navigate to.
4. j ≥ 1. At least one sham substitution restores a reading the log records as considered and rejected. (If this holds, at least one "quality-neutral" edit is a documented reversal of a documented choice.)
5. The four constraints will be satisfiable at 8 sites. (If they are not, that is a different and larger finding and §8 says what happens.)

7. Threats, stated in advance

• The lead authored both the log and the sham. Irreducible here; the project has no second translator. Mitigated only by the ordering (log frozen first, in its own commit), by the prohibition of §4.3, and by the measurement being mechanical. The direction matters and is favourable: residual knowledge of the log biases the lead toward avoiding D, which pushes k down — against predictions 2 and 3. A "FAILS" verdict is therefore reached despite the bias, not because of it. A "SURVIVES" verdict would be badly confounded and §8 says so.
• One passage, one translator, one language pair. Nothing here generalises to how a sham behaves on a published reference, which is the case Tier D actually runs — and published references have no logs, which is precisely why the question could not be asked there. What transfers is the mechanism, not the number.
• The decided-token set is the log's, not the translation's. A decision the lead made and did not write down is invisible, so c is a lower bound on the truly-deliberated fraction and k is measured against a conservative D. This too pushes against prediction 3.
• Anchor length is a free variable the lead controlled. Long anchors inflate c mechanically. Countered in part by the log's anchors having been written to be unique substrings rather than to be long, and by §9's sensitivity analysis, which recomputes everything with every anchor truncated to its first three tokens.
• §6.5's four constraints are interpreted by the lead. "No sense shift" and "free variation only" are not decidable by machine. Each of the 8 is argued in sham.md and can be disputed on the record.

8. Failure criteria — what voids or qualifies this run

• Voids it: opening the translator's log between the two freezes; editing any anchor after the freeze; adjusting the null after seeing k.
• Qualifies it: if fewer than 8 constraint-satisfying sites can be found, the run reports the number found and reports no k; that outcome is a stronger version of the same finding and is recorded as such rather than being converted into a pass.
• A "SURVIVES" verdict is reported with the confound named in the same sentence — the lead knew the log existed and may have avoided it — and licenses no more than "on this passage, a lead who has recently written a decision log can place eight sham sites away from it."

9. Verification

analysis/verify.py recomputes c, k, j and the null distribution from translation.md and sham.md alone, importing nothing from analysis/measure.py and nothing from tools/, with its own tokeniser and its own null sampler. Amendment A11 of E-20260726c-forced-or-borrowed-ru and the reason given in RS-20260726c-name-tokens-repair: a 218-check verification pass once found nothing while a shared function was broken in three ways, because both implementations called it.

It must additionally:

1. assert each of the 25 anchors occurs exactly once in the English;
2. assert each of the 8 sham spans occurs exactly once, and that applying all 8 to the frozen English yields the sham text byte-for-byte;
3. assert the sham text's word, sentence and paragraph counts equal the original's — §6.5's constraint, checked rather than asserted;
4. recompute the null by exhaustive enumeration where the space is small enough and by an independently seeded sampler otherwise, and report both;
5. recompute everything under the three-token anchor truncation of §7 and report c, k and the verdict under it.

If the truncated-anchor sensitivity analysis flips the verdict, the verdict is reported as flipped and neither version is preferred; that is what a sensitivity analysis is for.