6. The jury that could not be calibrated

The charter's fourth commitment was "calibration precedes authority": a panel of outside models could score translations, but its scores would carry no weight until it had passed a test. This section is the story of the four attempts, told from the panel page and the verdict pages, and of what the uncalibrated jury measured anyway.

The first design: reproduce the critics

The original gate asked the jury to reproduce documented critical consensus — to prefer, blind, the translation that critics and a prize had preferred. It ran once, on July 25, on the two English Botchan chapters the owner had supplied. On the real test (a 1972 version against the prize-winning 2005 one) record-fit was 0.55 on voice, 0.42 on affect, 0.43 on literary quality; only naturalness reached 0.80, and that was "register-cued." The jurors recognized the work on 93 per cent of probes. The verdict: "panel calibrated on NO sense." The charter's amendment of that day retired the design as a gate and kept it as a certification, which was run twice more on Garnett's and Hapgood's Turgenev and never certified: three seats identified both translators from a few hundred words ("the instrument is not blind"), and under a false attribution the jury held the record's accuracy verdict harder (23 of 24) while its verdict on English style "falls from 20 of 24 to 14 of 24, a coin."

Tier D: detect the damage

The replacement asked a question that could be powered at will: can the jury detect a translation deliberately damaged in one specific respect, and say which respect? Damage came from operators in a published catalogue of translation failure — Berman's tendances déformantes — never invented by the lead, since that "would test whether the jury detects lead-shaped damage." A pass required two controls: a sham arm, a quality-neutral edit of equal size on which the jury had to sit at chance, "what makes a positive result mean anything"; and a held-out arm, two independent competent translations, chance expected. The headline metric was specificity: when accuracy is damaged, the accuracy score must fall further than the untargeted senses. "A jury that says 'worse' uniformly detects damage but is not sense-calibrated."

July 25. The first run found the shape of the problem: the jury "cleanly separates damaged from undamaged and then cannot localise." Detection was six of six; the drop on the targeted sense was 1.17 against 1.08 on each untargeted one. And it scored one kind of damage — stripping a text of its marked foreign features — as an improvement: "the jury rewarded exactly the move a major theorist spent a book calling the central deformation of translation."

July 26. The first complete run with all three controls: not passed. The sham broke into its lower branch — in none of six units did the jury prefer the unedited text — and the sham's own decision rule turned out to fire half the time under the null. The instrument was rebuilt over ten days.

August 2. The run under the repaired rules was "the strongest run this project has produced." Detection was at ceiling — twelve of twelve units in both damage cells — and the sham sat cleanly in band. It failed on one pre-registered number: at eight damaged sites in a passage of 300 to 440 words, the naturalness score fell by 1.12 when the rule allowed 0.75. The verdict's gloss: "Eight accuracy errors in a 299–441-word passage do not read to this jury as eight accuracy errors" — they read as English gone wrong. The redesign that would have made the lighter dose primary needed a decision the project's own rule reserved to the owner, and he was not asked: for 157 sessions the "For Tom" block said nothing needed his attention, and every jury score carried the word provisional.

September 6. Asked on September 4, the owner authorized one redesign, on the condition that a second failure would end the approach. The design declared the light dose primary, stated specificity as a ratio, used fresh materials, passed an outside critic, and was ratified by a later session. It ran on a fresh day — ninety-six calls, none failed, $0.73 — and failed on two independent gates. The sham broke, for the first time into the upper branch: the jury preferred the unedited text at nine of fifteen units although every sham edit was a same-register paraphrase with no change of content. And at the primary dose, detection did not fire — nine of twelve, with one unit preferring the damaged text, where the rule allowed none. The verdict page offers "the most likely reading" rather than a proof: the design had shown the jury no source text except in the positive control, and a jury asked to score fidelity without the source "cannot be assumed to be measuring that sense." It measured consistency — internal self-contradiction — which moved more than accuracy at both doses. The run had also swapped one juror, and the page cannot separate the two explanations. The panel page records the state as FINAL: "TIER D CALIBRATION IS EXHAUSTED. NOT CALIBRATED, permanently, not pending recalibration."

What the uncalibrated jury still measured

The provisional flag did not stop the jury being used; it stopped its scores being cited as evidence of quality. Used descriptively, it fixed its own interval: gross damage moved scores by about 4.3 points on a seven-point scale; removing a translation's craft by 1.5; two competent translations differed by 0.17 to 1.33; the same juror re-scoring the same text on another day moved by 0.23 to 0.25. When the project's own translations were finally scored blind on August 3, five of them fell between 5.0 and 6.7 and only naturalness separated them. Re-scored 159 sessions later, the scores held — a mean shift of 0.127 against a retest floor of about 0.23 — and a check a pre-run critic had added caught one juror scoring nine cells in ten at the top of the scale.

Two descriptive findings survived every later narrowing. A self-revision pass by the same model lifted all six senses within 0.134 of one another — not the "naturalness without accuracy" shape the framework's one candidate rule had predicted, and the rule was struck. And foreignizing a passage of Lu Xun was not a trade-off but "a one-way expenditure": it cost 5.5 of 7 naturalness points, moved accuracy by nothing, and bought 2.8 points of perceived source carriage, a shape that replicated on Kleist.

What it means

The whole calibration line cost about eight dollars in API fees; money was never what stopped it. A calibrated jury would have let the handbook say better. Its absence means that every sentence in this essay of the form "translators do X" is a count, every sentence of the form "X costs Y" is a measurement on a jury whose relation to human readers is unknown, and the sentence "X is better" does not appear. The project's last word is in the handbook's evaluation entry, which states that "the jury is not, and will not be, calibrated," and gives a translator the honest alternatives: an anchor, a precedent, a declared purpose, and a reader.