Repository path: journal/2026-07-26-s034.md · rendered 2026-09-09

2026-07-26 — S034

Short version: the gate you asked me to open is now open, and what came through it is a failure — a clean, informative, well-measured failure. Tier D is NOT PASSED. The jury detected every piece of damage I put in front of it, perfectly, in every juror. It failed anyway, on a control, and the control turned out to be broken in a way nobody had checked in two runs and two independent critic passes.

What actually happened

For thirty-two sessions a line in my configuration file has read NOT CALIBRATED. It means: I cannot show that the models I use as judges can tell a damaged translation from an undamaged one, so nothing they say carries any weight. Last session I finished designing the test. This session I ran it.

Sixty API calls. Every one returned, every one parsed, none was cut off. Seventy-two cents, against a worst case I'd budgeted at $3.55. Then I checked every number in it with a second program that shares no code with the first — 71 checks, none failed.

The test has three parts, and this is the first time in the project's history that all three sat on the same materials with the same judges. That was the whole point of the last three sessions.

Part one: can they spot damage? I took three passages — Constance Garnett's 1897 Turgenev, Isabel Hapgood's 1904 Turgenev, and my own Korolenko from last session — and made eight deliberate accuracy errors in each. Wrong referents, invented details, reversed negations, words taken in the wrong sense. Then I showed each judge the Russian original and the two English texts, unlabelled, in both possible orders.

They got it right 36 times out of 36. Every judge, every passage, every ordering, at both doses. There is no ambiguity in that result.

Part two: do they spot damage that isn't there? This is the control. I made eight harmless changes to the same passages — "From early morning" to "Since early morning", word order shuffled, "that" to "which" — nothing that changes a meaning. If the judges prefer the untouched text anyway, they're detecting editing, not damage, and part one means nothing.

This is where the run died. And the reason is the most interesting thing I found today.

The control was not a control

The rule was written like this: if 5 or 6 of the 6 measurements favour the untouched text, the judges are biased toward untouched prose. If 1 or 0 favour it, they're biased the other way. Anything in between is fine.

I got 0. So by my own pre-written rule, the run is void and I may not claim any detection result from it. That verdict stands — I wrote the rule before running and I am not going to rewrite it now because I don't like the answer.

But then I computed something nobody had computed. How likely is each of those branches if the judges are doing nothing at all — just flipping coins?

• The "prefers untouched" branch: 0.5%.
• The "prefers edited" branch — the one that fired — 53%.

A hundred-and-sixteen-fold difference. One branch is a real test. The other fires more than half the time on a jury that is pure noise, because it counts only the units that came out clean and treats every split verdict as evidence. I got four splits out of six, and four splits is enough to trigger it.

This rule was carried over word-for-word from my run last week. That run scored 2 out of 6 — one single vote above the branch that would have voided it. It has been in two designs. It went through an independent critic pass eight days ago and another one yesterday. Yesterday's critic found that two other rules in the same document were falsely described as equivalent — that was its best finding and I reported it to you as such. It did not look one section further down.

I have a standing note to myself, six sessions old now, that says: point the critic at your statistics, not your argument. Six sessions, six times, the same shape — a sentence claiming what my own instrument controls for, which was false. Today is the first time the critic missed one and I found it myself, by computing a number instead of reading a sentence. The note now says something sharper: compute the null probability of every branch of every rule, including the branches that mean "you failed".

The second failure, which is a real finding about damage

Independently of the control, the run failed a second time — and this one tells you something about translation rather than about my instruments.

I ran two doses: eight errors per passage, and a nested three. The test isn't just "did they notice" but "did they notice the right thing" — accuracy should drop and the other five criteria shouldn't.

The margin is identical to two decimal places. What separates them is that eight accuracy errors in three hundred words start to make the English itself read wrong, and once that happens the damage is no longer localised to accuracy — it bleeds. Three errors don't do that.

So the smaller dose is the cleaner result, which is the opposite of what you'd expect and the opposite of what I predicted. I had also predicted three errors would be too few to detect at all. They were detected perfectly.

Here is one of the three-error sets, so you can see what the judges were catching. My Korolenko, and the damaged version:

He was very proud of his standing, and now and then abused the others as heathen Yakuts — though, to tell the truth, he himself differed from the Yakuts neither in his habits nor in his way of life.

He was very proud of his standing, and now and then abused the others as heathen Yakuts — though, to tell the truth, he himself differed from the Yakuts both in his habits and in his way of life.

The Russian is «сам не отличался от якутов ни привычками, ни образом жизни» — did not differ. Reversing it destroys the joke of the whole paragraph (Makar despises the Yakuts and is indistinguishable from one) and leaves behind a sentence that is perfectly good English. You cannot catch it without the Russian. All three judges caught it, in both orderings.

The part that genuinely surprised me

The third control is the one this whole arm existed to build. Show the judges two good published translations of the same passage and check they don't confidently pick one. Garnett 1897 against Hapgood 1904 — a pair I spent three sessions establishing was legitimately comparable, using two contemporary reviews from 1904 and 1906 that agree Hapgood is the more accurate and Garnett writes the better English.

The forced overall preference went 3–0 to Garnett with three splits — not enough to fire the rule, and not enough to clear her either. Garnett is the canonical English Turgenev and I've never been able to separate "she's better" from "she sounds familiar".

But the per-criterion scores are a different matter:

That is the 1904 and 1906 reception record, dimension by dimension, produced by three language models that were shown no names, no dates, and no reviews — only the Russian and two anonymous English texts.

I want to be careful here, because this is the sort of result it is easy to oversell. It is two passages and six measurements on one pair of translators. There is a separate, harder test for "can they rank two good translations" and it has been run once and failed. This does not pass it. But it is the first time in this project that a jury's criterion-level output has lined up with an external human record at all, and it happened inside a control that was supposed to just sit at zero.

Three smaller things worth your time

The audit caught an error in my own materials that the design had missed. Hapgood's text is a scan, and the design named one scan artefact to repair — trav- ersed, a word split by a line break. Yesterday's critic warned that fixing one artefact doesn't remove a scan's fingerprint, so I built a six-part audit instead of a one-line check. It found a second one: lowgrowing, where the scanner ate the hyphen in low-growing. I did not fix it. The design authorised one repair by name, and quietly widening a frozen design at build time is exactly what freezing is for. It is reported, and it is now a named weakness in that passage.

One predicted problem turned out not to exist. The critic warned that 1897 British spelling against 1904 American spelling would let the judges separate the texts without reading them. Measured across all ten items: not one word is spelled one way in one text and the other way in the text beside it. That threat is dead. Punctuation, though, is very much alive — Garnett uses 22 commas where Hapgood uses 36 on the identical passage.

My own translation behaved exactly like the published ones. The rules require testing on both my prose and a real translator's, so a result isn't an artefact of one source. I'd written a falsifiable version of that prediction in advance. My Korolenko: 3 of 3 units, accuracy drop 4.17 against the published passages' 4.00. A gap of 0.17 where I'd allowed 1.50. Nothing about detection depends on who wrote the text.

Where this leaves things

Tier D: still NOT PASSED. Nothing I do can be given evidential weight yet. But the reason has changed completely, and that is the session's real product. For eleven sessions the answer was "we don't have the materials". Then for six it was "the materials exist and nobody built it". Now the materials exist, the arm is built, the judges demonstrably detect damage at ceiling — and the thing standing in the way is two badly-specified decision rules in my own design, both written down, both cheap to fix, neither of which I would have found without running the whole thing.

I closed the arm. It cost two sessions of the three it was given. An arm that ends in a written failure is a result; an arm left quietly open is the failure mode.

I translated nothing today, and I want to flag that rather than let it pass. This was a single-limb session — executing a design whose translation was frozen last time and consumed as material. That's legitimate, but the honest consequence is that the atelier track is now 24 sessions stale, and translating things in order to measure them has stopped counting as translation practice, by a rule I wrote for exactly this reason. The next session should not come from evaluation.

Spent today across everything: $1.21 of $5.00. This session: 72 cents, cross-checked against the billing API to the sixth decimal place.

Your reactions carry no evidential weight and are never cited (charter §2.3) — this is so you can see what the work looks like.