Repository path: journal/2026-09-06.md · rendered 2026-09-09

2026-09-06

This is a long-running study of literary translation: I translate public-domain fiction myself under controlled conditions, have outside AI models judge or measure the results blind (I never grade my own work), and try to distill what survives into a practical handbook, organized by translation problem.

Where things stood

For months, the project's biggest open question has been whether the outside AI models I use as judges can actually be trusted to judge. The test is simple in principle: take a competent translation, deliberately damage it with a few real errors, and see whether the judging panel can tell the damaged version from the clean one — reliably, and specifically because of the errors rather than for some other reason. This test has never passed. Two earlier attempts (in late July and early August) both found the panel detecting damage well, but failing a second, stricter requirement: the panel also had to mark down an unrelated quality (how natural the English sounds) by too much, which suggested it was reacting to "something's off here" in general rather than to inaccuracy specifically. The project's owner, Tom, authorized exactly one more redesigned attempt on September 4, with the understanding that if this one also failed, the approach would be considered exhausted and the handbook would keep resting only on what individual translation sessions find by hand, never on a panel's verdict.

What this session did

I ran that one authorized attempt, start to finish. First, an outside AI model reviewed the design adversarially for flaws, and a second, different outside model cast the deciding vote on whether to proceed — both approved it as written. Then I ran the actual test: 96 separate judging calls to three outside models, at a total cost of 73 cents, with zero failed calls. I then independently recomputed every number the test produced, from the raw responses, to make sure nothing was taken on faith.

The test failed, and it failed in a new and more informative way than before. Two things went wrong, either one enough on its own. First, a control designed to catch a panel that penalizes any edited text (regardless of whether the edit actually changes meaning) tripped for the first time in four attempts: shown a passage edited only with harmless, meaning-preserving rewordings, the panel still preferred the untouched version 9 times out of 15. Second, at the dose of damage meant to be the test's centerpiece — three real errors scattered through a few hundred words of otherwise clean prose — detection didn't fire reliably; one of the four test passages fooled the panel outright.

The more interesting finding is why, and I think I can show it rather than just assert it. This redesign deliberately withheld the source-language original from the panel in every stage except one, for a defensible reason unrelated to this problem (it was protecting a different measurement elsewhere in the test). But that choice turns out to matter a great deal: of the four types of error I planted (a wrong name, a wrong word, an invented detail, a dropped "not"), three of them are only detectable as errors by comparing against the source — without the source, an invented detail just reads as an odd but plausible detail. Only the fourth type (a dropped "not," which usually creates a direct contradiction with the surrounding sentence) is catchable from the English alone. And that is exactly the pattern in the data: every test passage whose random sample of three errors happened to include a dropped negation was caught perfectly; the one passage whose sample didn't include one was the passage that fooled the panel. In other words, without the source text in front of it, this panel doesn't appear to be measuring "is this accurate" so much as "does this contradict itself" — a related but narrower thing.

What it means

Per Tom's standing authorization, this was the one redesign attempt, and it failed. The model-panel calibration effort is now closed for good — not paused, not pending a future redesign. The handbook of translation guidance will continue to be built from what I find by hand doing the actual translation work, evaluated against published criticism, never from a panel's numerical verdict. Every quality score anywhere in this project's records stays labeled as unverified and provisional, permanently. This does not mean AI panels can never judge translation quality in principle — only that this project's own four attempts, across roughly six weeks, could not build an instrument that reliably and specifically detects accuracy damage, and no further attempt is authorized.

What's next

The next session returns to ordinary translation practice: rendering the opening of War and Peace into French, to test a specific gap in the project's craft notes that no existing guidance covers. The session after that continues writing the handbook's chapter on level of formality and address.

What needs you

Nothing needs your attention. I wanted to flag plainly that this closes out the redesign you authorized on September 4 — it did not pass, and per your own condition for that authorization, I am not attempting a further redesign.

Later the same day: translating the opening of War and Peace into French

A second session ran later on September 6 and did the ordinary translation work forecast above.

The problem. Tolstoy's Russian aristocrats speak French to each other — long stretches of the opening scene are written in French even though the book is in Russian, and Tolstoy translates his own French back into Russian in footnotes, for the Russian readers of his own day who didn't know French. I'd already translated this same opening into English once before (in early September), and found that every published English translator turns Tolstoy's French-and-Russian mixture into plain English, losing the fact that the characters are switching languages at all. That earlier session flagged one case my craft notes had no advice for: what happens if you translate the book into French — the very language the characters are already partly speaking?

What I did. I translated the same two opening chapters into French myself, from the Russian, without looking at any published French version first. The interesting part is what happens to the passages that were already in French: translating "into French," those passages don't need translating at all — I could carry a number of them into my French version completely unchanged, letter for letter. That never happens when the target is English. Where Tolstoy plants a Russian word inside a character's French sentence — for instance, when a scheming courtier calls himself someone's "мой верный раб" (in Russian, meaning "faithful slave") in the middle of an otherwise French sentence — I could, for the first time, leave the Russian word visibly foreign (in the original Cyrillic, with a quick French gloss next to it), because now Russian truly is the odd one out. In English, that same Russian word just disappears into ordinary English along with everything else.

Here's a taste of the translation — the novel's opening line, spoken by the hostess of a society party, with the French exactly as Tolstoy wrote it and my new French narration around it:

Eh bien, mon prince. Gênes et Lucques ne sont plus que des apanages, des поместья [domaines], de la famille Buonaparte... Eh bien, bonjour, bonjour. Je vois que je vous fais peur — asseyez-vous et racontez-moi tout.

Ainsi parlait, en juillet 1805, la célèbre Anna Pavlovna Scherer, demoiselle d'honneur et proche de l'impératrice Marie Feodorovna, en accueillant le prince Basile, personnage important et haut placé, le premier arrivé à sa soirée.

Then I checked my work against the one real French translation of this book old enough to be free to read (from 1879, by a Russian noblewoman writing under the pen name "Une Russe"). I expected her to have made the same choice I did — since Tolstoy's own French didn't need translating, why not just keep it? She didn't. She rewrote almost all of it in her own words, including the passages that were already correct French, and even quietly corrected a small misspelling of a real person's name (Lavater) that Tolstoy's own French had gotten wrong and that I had deliberately kept. The one place she does preserve Tolstoy's actual wording almost untouched is an actual letter quoted in the story — and even there she changes one detail (the hour of the party). And at the hardest single joke in the whole passage — a barely-literate estate manager who misspells the Russian word for "slave" and then spells out his own misspelling letter by letter — she didn't leave the joke in Russian either; she invented a matching misspelling in French ("esclafes" for "esclaves"), the same kind of trick an English translator has to use, even though French, unlike English, didn't actually need the trick.

So the honest finding is more interesting than I expected going in: having the option to just keep the author's own words doesn't mean a real translator takes it. The one real French translator measured chose to make the whole book sound like her own writing throughout, and treated an actual quoted document (the letter) as the one thing worth preserving verbatim — which lines up with a rule the project had already found elsewhere: translators reliably keep quotations of an actual document, and reliably don't feel bound to keep an author's own incidental wording. That rule, plus this session's example, both went into the relevant page of the growing handbook.

What needs you: nothing.