Repository path: workshop/experiments/E-20260728c-length-matching/design.md · rendered 2026-09-09

E-20260728c — what a length-matched control costs, and what a stratified one recovers

FROZEN BEFORE ANY MEASUREMENT AND BEFORE ANY TRANSLATION. Committed to git before the first English word of the translation limb was written and before any number in limb A was computed. The commit is the freeze.

Unit: S046, principal unit ARM-framework (T5), step 3.

0. The question, and why it is a fork rather than a task

RS-20260727c-arm-identifiability §11 item 1 prescribes a repair and offers two forms of it:

Match the control arm on length. The repair for §6 is a TEMP-equivalent control whose two texts do not differ systematically or randomly in length — or, failing that, a control analysed within length-sign strata rather than pooled.

ARM-framework's next step is to write that specification. The two forms are not interchangeable and the arm page does not choose between them; it says only that the second is "cheaper and probably better," which is a guess. Writing a specification by picking the option that sounds better is what note (abk) calls guidance that decides nothing, and it is exactly the kind of unevidenced prescription this arm exists to stop producing.

So this session prices both options before writing the rule:

• Option M (match). Build the control's two texts so they do not differ in length. Nobody has ever done this, and its cost is unknown. Length in a translation is not a knob: it is an outcome of every choice the translator makes. Constraining it is therefore an instruction to the translator, and an instruction changes the arm. Whether it changes it a little or a lot is measurable.
• Option S (stratify). Analyse within length-sign strata, never pool. Requires no instruction to anyone and no threshold. But it needs both strata populated, and it spends n.

Limb B prices M by carrying it out on real prose and counting what it costs. Limb A prices S by computing what it recovers at the n this project's paired designs actually have. The specification adopts whichever survives; if neither does, it says so.

The wire, in one sentence. The translation limb prices the match option by doing it and counting what it costs the arm, and the study limb prices the stratify option by computing what it recovers at the project's real n — and the control-arm specification is written from whichever survives.

Nothing here is a judgment of any translation. The jury is NOT CALIBRATED. S010's stored verdicts are used only to characterise S010's design, a use that can weaken a claim and never support one — the same restriction RS-20260727c operates under. The lead's three Kleist texts are never evaluated: the only quantities taken from them are word counts and a token diff of two of them against each other. That is why a work with unmeasurable contamination is admissible here (manifest §Comparator reachability), and the sentence is load-bearing: a remembered phrase would change none of the numbers this design computes.

1. Limb A — what stratification recovers, from S010's stored data

Materials, all already in the repository. workshop/experiments/E-20260724-r01r02-selfrevise/runs/: items.json (48 payloads), blinding-key.json, jury/P2/*.json and jury/P3/*.json. Twelve TEMP items and twelve MAIN items, each judged by two jurors in two presentation orders — four votes per item per sense.

Frozen extraction, verbatim (note (r)).

1. Word count of a passage = len(text.split()), Python 3, on the payload strings p1 / p2 exactly as stored. wc -w is not used (note (abl)). Counts are recomputed from items.json, not read from E-20260727b's identifiability.json, so the sign convention is this design's own.
2. Δ for an item = (words of the treatment passage) − (words of the baseline passage), where the roles are read from blinding-key.json: on MAIN, treatment = revision, baseline = d7; on TEMP, treatment = d4, baseline = d7. Δ is computed once per item from the _ab payload; the _ba payload is the same two texts and must give the same |Δ| — asserted as a check, not assumed (note (vv)).
3. A vote is parsed from the juror response body: strip a ``json fence if present,json.loads, read[sense]["winner"], which is the string"1"or"2", and map it through the payload'spassage1_role/passage2_role. Any response that fails to parse, or whose winner is neither"1"nor"2"`, is dropped and counted; the count is reported.
4. pref_longer(item, sense) = (votes for the longer of the two texts) / (votes cast for that item and sense). Items with Δ = 0 are excluded from pref_longer and counted separately.
5. pref_treatment(item, sense) = (votes for the treatment role) / (votes cast).
6. Senses: the five in the front matter, in that order.

A1 — is the length response graded, or sign-only?

The distinction decides Option M by itself. If the jury responds to the magnitude of a length difference, matching to a tight tolerance shrinks the effect and M works. If it responds only to the sign, then any nonzero difference delivers the full effect, no achievable tolerance helps, and M is worthless unless matching is exact.

• A1a. Spearman ρ between |Δwords| and pref_longer, over the 12 TEMP items, per sense and pooled over the 60 (item, sense) cells. Ties by midrank. ρ reported with its n; no p-value is attached to the pooled figure — the five senses are judged on the same items by the same jurors out of the same response object and are nowhere near independent (the constraint RS-20260727c §6 imposes on itself).
• A1b. The same for MAIN.
• A1c. The decisive descriptive: rank the TEMP items by |Δwords| and report pref_longer for the smallest-|Δ| item and the largest, per sense and averaged.

Pre-registered reading, both branches (note (abh)). - GRADED iff pooled |ρ| ≥ 0.40 on TEMP and the smallest-|Δ| item's mean pref_longer is below the TEMP median. → the specification may permit M with a stated tolerance. - SIGN-ONLY iff pooled |ρ| < 0.40 on TEMP and the smallest-|Δ| item's mean pref_longer is at or above the TEMP median. → M is admissible only as exact matching. - INDETERMINATE otherwise, and this branch is expected to be reachable: at n = 12 a Spearman ρ is a weak instrument, and the design says so in advance rather than after seeing it. An INDETERMINATE result licenses nothing about M and must not be reported as evidence for S.

A2 — the tolerance question

Report the full |Δwords| distribution for TEMP and MAIN: min, median, max, absolute and as a percentage of the baseline text's length. This is descriptive and is registered because the specification needs the number: a tolerance rule cannot be written without knowing what differences real paired arms actually show.

A3 — what stratification recovers at n = 6

For TEMP, split the 12 items by the sign of Δ and compute, per stratum and per sense, pref_treatment and an exact Clopper–Pearson 95% interval on the underlying vote counts (6 items × 4 votes = 24 votes per stratum per sense). Report the interval width. Also report a per-item jackknife SD of the stratum mean, because the 24 votes are 6 clustered units and the binomial interval will be too narrow — both are reported and the wider one governs.

Pre-registered failure criterion. If the jackknife-widened interval on either stratum spans 0.5 by more than ±0.15 on a majority of senses, the stratified estimate at n = 6 is uninformative, and the specification must carry a minimum-n clause computed from this. If it does not, S is usable at n = 6 as it stands.

A4 — the degenerate case, which is the one that matters

MAIN is 10 / 12 one sign (RS-20260727c §4). Run A3's computation on MAIN and report the per-stratum n. A stratification rule is only as good as its smaller stratum, and the treatment arm of a regime comparison is exactly where one stratum collapses. Report what the rule yields at n = 2, and state the rule that follows for an empty or near-empty stratum.

Limb A predictions, registered before any number is computed

2. Limb B — what matching costs, priced on real prose

Material: workshop/canon/bettelweib-locarno/source.txt — Kleist, «Das Bettelweib von Locarno» ¶1–3, 361 German words, 1810 first printing. Manifest for provenance, PD status, the comparator-reachability check and the selection rationale.

Procedure. Each step is committed before the next begins; the commit is the freeze.

1. T-bettelweib-locarno-R06-v1 — regime R06 v1.0, lead single pass, source only, no return pass. Translator's log written as the draft is written. Commit. Word count W_A.
2. T-bettelweib-locarno-R04-v1 — the ordinary R04 self-revision of that draft against the source. No length target is set, considered, or computed at this stage; W_A is not looked at. Log continues. Commit. Word count W_B.
3. The matching pass — take the R04 revision and bring its word count to exactly W_A, changing as little as possible, and never changing anything for any reason other than the count. Every edit is logged with the count it moved. Filed as T-bettelweib-locarno-R04m-v1. Word count W_C.

R04m is deliberately NOT a numbered regime and no page is created for it in workshop/regimes/. It is defined here and only here, because one live outcome of this experiment is that a length-matched arm must never be used as a control — and promoting it to the regime shelf would be building the thing the result may forbid.

Order matters and the cost of the order is stated. The matching pass is applied to the free revision, so step 3 is primed by step 2 by construction. That is not a defect: it is what a real matched-control procedure would do — you write the arm, then you match it. What the order does forbid is any claim about what an independently written length-matched revision would look like, and no such claim is made.

Frozen instrument, verbatim (note (r)).

• Tokens of a translation = re.findall(r"\S+", body) where body is the text between the ## Translation heading and the next --- line, with markdown emphasis markers * and _ retained (they are part of no word boundary here) and blank lines collapsed. Word count = len(tokens).
• Diff = difflib.SequenceMatcher(None, free_tokens, matched_tokens, autojunk=False). An edit site is one maximal opcode block that is not equal. sites = their count; tokens_changed = sum(max(i2-i1, j2-j1)) over sites.
• The revert test (B4), three-way and all three branches pre-registered. For each site, build the anchored run by extending the changed span with up to 3 equal tokens on each side: F* from the free text, M* from the matched text. Then
• REVERT iff M* occurs as a contiguous token run in the R06 draft and F* does not;
• NEW iff neither occurs in the draft;
• AMBIGUOUS iff both occur.

Anchoring is what makes the test survive pure insertions and pure deletions, where one of the two raw runs is empty and would match trivially.

Measurements. B1 Δ_free = W_B − W_A, absolute and as % of W_A. B2 sites and tokens_changed. B3 whether W_C = W_A exactly. B4 the REVERT / NEW / AMBIGUOUS counts. B5 Metric A on this pair — recorded because RS-20260727c §11 item 2 requires it of every future paired comparison, and reported as undefined at n = 1, with the sign alone given. Recording an undefined statistic and saying so is the point of the requirement.

Limb B predictions, registered before the first English word

Registered failure criteria, stated in the direction that would embarrass the lead.

• If B2 = 0, matching cost nothing, Option M is cheap, and the specification must say so and must not prefer S on cost grounds.
• If B4 = 0 REVERTs, the claim that matching reverses the treatment is not supported by this run and may not be asserted anywhere, in the specification, the result page or the journal. A NEW-only outcome means matching perturbed the arm without reversing it, which is a weaker and different finding.
• If P7 fails large (|Δ_free| > 10%), the free revision moved length so much that matching is a major intervention, and that is a finding about R04 rather than about controls.

n = 1, and what that costs. One passage, one language pair, one translator, one matching pass. This cannot estimate how often matching binds. What it can do is settle an existence question — does a length constraint reach the translator's choices at all, and does it reach back through the revision? — and existence questions are answerable at n = 1 in the direction of "yes" and not in the direction of "no". A zero result here is weak; a nonzero result is not. That asymmetry is why the failure criteria above are written so that the zero branch forbids the claim rather than licensing its opposite.

3. Gates and hygiene

• Pre-run critic. One independent non-Anthropic model, pointed at the statistic definitions and at every branch of every pre-registered rule (notes (rr), (abh)) — including the branch probabilities and the power (note (bci)). Findings recorded in critic.md with accept/reject and written rationale, before any of §1 is computed and before the translation begins.
• Post-run verification. An independent verify.py that imports nothing from tools/ and reimplements the tokeniser, the diff-site rule, the revert test, the Spearman and the Clopper–Pearson bound from this text (note (xx)). It asserts the expected count of every unit extracted (note (vv)): 48 payloads, 24 TEMP votes per stratum per sense, 3 stored paragraphs, 3 translation artifacts.
• Cost. Limb A and limb B are local computation and lead translation: $0. The only spend is the critic pass.
• Contamination. Declared on each artifact as suspected, UNMEASURED, with the manifest's three verified reachability checks as the basis and the reason the gate cannot run.
• No evaluative claim about any of the three translations appears in any output of this experiment. Any sentence in a translator's log that is evaluative carries internal-judgment-only.

4. Amendments accepted from the pre-run critic — made BEFORE any measurement and before the first English word

critic.md records the pass (openai/gpt-5.6-terra, verdict NEEDS-REDESIGN, nine findings) and the disposition of each. The text above is left standing so the amendments are auditable; where §4 conflicts with §§0–3, §4 governs.

4.1 B2 was true by construction — the finding of the pass (G7)

"A required exact-count editing pass necessarily reaches some choice whenever W_B differs from W_A." If the count must move, at least one site must change: B2 > 0 has a false-alarm rate of 1, and P8 as written was near-vacuous.

B2′ = tokens_changed − |Δ_free| replaces it as the primary quantity: the tokens the matching pass moved beyond what the arithmetic demanded. A pure trim — delete exactly the surplus words and touch nothing else — scores B2′ = 0. Every token above zero is a change the constraint forced without requiring.

• P8 (re-registered): B2′ ≥ 5 tokens. If B2′ = 0, matching was a pure trim, Option M is cheap on this evidence, and the specification must say so.
• sites and tokens_changed are still reported, now as descriptive companions to B2′ and not as evidence of cost.

4.2 The revert test is made total, and its claim is narrowed to string recurrence (G3, G8)

Four classes, mutually exclusive and exhaustive. For each site, with F* the anchored free run and M* the anchored matched run, and occurs(X) = "X appears as a contiguous token run anywhere in the R06 draft token list" (existential — no location-selection rule is needed, which dissolves the repeated-occurrence objection):

Anchor construction, exact. For an opcode block (tag, i1, i2, j1, j2): L = 3 tokens of left context taken from the equal tokens immediately preceding, truncated at the start of the text; R = 3 likewise following. F* = free[i1-L : i2+R], M* = matched[j1-L : j2+R]. Left and right context are identical strings in both texts by construction (they come from equal blocks), so F* and M* differ exactly at the changed span. Pure insertions (i1 == i2) and pure deletions (j1 == j2) are handled by this without special-casing, which is the reason for anchoring.

What a REVERT licenses, and it is less than the original text claimed. It licenses "the matched wording recurs in the draft and the free wording does not" — string recurrence, not semantic reversal of a treatment. An empirical coincidence floor is computed (§4.4) rather than assumed.

Reproducibility. difflib.SequenceMatcher(None, a, b, autojunk=False); the Python version is recorded with the result; verify.py reimplements the classifier from this text and runs reference fixtures covering replacement, pure insertion, pure deletion, a site at the start of the text, and a run repeated twice in the draft.

4.3 A3 gets a cluster-valid interval and an unambiguous inequality (G2)

Clopper–Pearson on 24 votes assumes 24 independent trials while the design itself says they are 6 clusters of 4; and "the wider one governs" is not a procedure.

• Estimand: the stratum mean of the item-level pref_treatment values (6 numbers, one per item).
• Interval: a cluster bootstrap over items — resample the stratum's items with replacement, 10,000 draws, seed 20260728, percentile 2.5 / 97.5. Cluster-valid by construction; no independence assumption.
• Failure inequality, exact and unambiguous: the stratum fails iff hi − lo > 0.30. No reference to 0.5 and no endpoint ambiguity.
• Clopper–Pearson on the 24 votes is still reported, labelled anticonservative, precisely so the size of the error the original design would have made is visible.
• P3 (re-registered): the bootstrap interval fails (hi − lo > 0.30) on ≥ 3 of 5 senses.

4.4 A1 measures association, not mechanism (G1, G6)

The GRADED / SIGN-ONLY / INDETERMINATE labels are demoted from an identification claim to an association claim, because item quality can correlate with |Δ| under a sign-only mechanism and noise can erase correlation under a graded one. The branch names are kept for continuity with §1 and are read as association present / association absent / undetermined.

• A null association does NOT establish sign-only, and the result page must not say it does.
• A permutation null replaces the undefined error rate: 10,000 shuffles of pref_longer across items within sense, two-sided p from the exact permutation distribution.
• The detection floor is reported with the result (note (bci)): the smallest |ρ| that would reach p < 0.05 at n = 12 under the realised tie structure. What the instrument could not have caught goes on the page.
• Restricted scope: A1 observes naturally-occurring length differences and cannot show that reducing a difference would shrink an effect. That sentence goes in §Not licensed.

4.5 The prediction table is sorted into predictions, fixtures and a bias declaration (G4, G5)

• P2 and P4 are not predictions. They are deterministic assertions about frozen files, either true or false conditional on the repository. They become fixture checks (note (vv)): asserted counts that fail loudly and are not scored as tests.
• P5 is removed from the prediction table. It is a bias declaration and is restated as one: the lead expects the specification to adopt S with a minimum-n clause. Recorded so that adopting M would count against it; not evidence.
• Limb B is a non-generalizable case demonstration. Hypothesis-testing language is struck from it. P6, P7, P8 and P9 are registered expectations about a single case, scored met / not met, with no false-alarm rate and no power claimed for any of them.
• The critic's own remedy — sample several passages and translators, define a population event, power the n — is right and is out of budget for one session. It is recorded as the successor step rather than pretended away.

4.6 What stratification actually controls, which is less than the original text implied (G9)

A3 measures precision, not validity. Stratifying on the sign of Δ removes the sign confound within a stratum by construction; it does not remove magnitude confounding inside the stratum. The specification must therefore carry a within-stratum magnitude check as a required companion to the stratification rule, and must not present S as controlling length confounding in general.