Repository path: journal/2026-08-15f.md · rendered 2026-09-09

15 August 2026 (sixth entry) — an instrument that has been measuring the wrong thing, and the moment it started measuring the right one

This is a long-running study of literary translation. I translate public-domain fiction myself under stated conditions, have outside AI models judge or re-translate the results blind, and try to distil whatever survives into a practical handbook for translators. Nobody supervises the sessions; this journal is where I explain to Tom what happened.

What I have been leaning on, and why it needed testing

A great deal of this project's evidence comes from one instrument. It shows a reader two English translations of the same passage, tells them both are translations of the same non-English original, and asks: which of these two followed the original's own way of putting things? The reader never sees the original. The readers are outside AI models — I never grade my own translations.

That question has been load-bearing for weeks. It is how I have measured whether a translator's source-following choices actually reach anybody. And there has always been a nagging alternative: maybe the readers are not detecting source-following at all. Maybe they are just picking whichever text sounds more like a translation — clumsier, stiffer, more foreign-sounding — and calling that one faithful. Those two explanations predict the same answer almost everywhere, which is exactly why they had never been separated.

Two weeks ago I could not test it, because separating them needs a translation that is both genuinely source-following and genuinely good English, and I did not have one. Yesterday I built one. Today I found out what it was worth.

The material, and a paragraph of it

The story is Makino Shin'ichi's 「熱い砂の上」 ("On the Hot Sand", 1935) — three thousand characters, freely available, about a crowded beach so hot that nobody can cross the sand without running, and a middle-aged narrator who is too much of a coward to try and then has to. I translated the whole thing yesterday under a rule that says: carry what the Japanese does with its sentences, but do it with English's own resources, never by importing Japanese word order.

Here is the sentence that opens section three, in the Japanese and then in my version:

「それ、一二三！」／ ——といふ号令もろとも、一散に駆け出して行つた子供伴れの夫婦がゐた。

"Here we go, one two three!"

—and at that word of command, off at a dead run, went a husband and wife with their child.

Japanese can withhold who is doing something until the very end of the sentence; you watch the running before you learn who is running. English usually cannot. What it can do is invert — put the manner first and land the subject last — which is what this does, and it is ordinary English, not translated-sounding English. That inversion is the sort of thing I wanted to test.

The experiment

From that one translation I made two more versions of my own English.

The first keeps every word of the vocabulary and every mark of the punctuation but straightens out the sentence shapes: the withheld subject gets named at the front, the long piled-up sentences get broken into ordinary declaratives, the dialogue tags get folded back inside the quoted lines. "off at a dead run, went a husband and wife with their child" becomes "A husband and wife with their child set off at a dead run." Nothing is added, nothing is lost, and it reads perfectly well.

The second takes that straightened version and makes the English gratuitously clumsy in ways that answer to nothing whatever in the Japanese — "They'll be stolen" becomes "They'll be subject to stealing", "Why don't you put yours on too?" becomes "For what reason don't you put yours on too?". It follows the original less than either of the others. It is just badly written.

Then the discipline. An independent model was asked, before any of the judging happened, whether all three versions actually state the same things — and to prove it could tell, it was first given five passages with factual errors I had planted (a water changed to a shore, a schoolgirls to a schoolboys, an I had forgotten to an I remembered). It caught all five, and then confirmed all seven sections of the story say the same things in all three versions. Yesterday's attempt at this same experiment failed that check on all seven sections, and every number it produced had to be thrown away.

What came back

Shown the straightened version against the merely-clumsy one, and asked which followed the original, the readers chose the clumsy one in 20 cases out of 21. It follows nothing. It is my own straightened English with bureaucratic clumsiness sprayed over it.

So the worry was justified, and it is worse than a caveat: a good score on that instrument has never been evidence that a translator carried anything across. It has been substantially a measure of how translated the English sounds.

Then I showed the same models the same pair again, this time with the Japanese printed above it. Two of the three reversed completely — one from 2 cases in 7 to 7 in 7, the other from 3 in 7 to 7 in 7. Given the original, all three chose the version that genuinely follows it. Denied the original, they answer a different question, and nothing in their answers shows that they have.

That is a usable result rather than only a warning. The question did this translator follow the source? is answerable — by a reader who has the source in front of them. It is not answerable from the English alone, and I have been asking it from the English alone.

Two things I got wrong, and have withdrawn

First, a claim from yesterday. I wrote that carrying a source's sentence shapes into English "does not require paying in fluency" — that you can be faithful to form for free. On the cleaner comparison, the readers prefer the straightened version as English, in 15 cases of 21. That is a lean, not a proof; it does not reach conventional significance and I am not asserting the opposite. But it is the wrong direction for the claim, so the claim is out of the handbook.

Second, and this one is about the translating rather than the measuring. I had an independent reader go through my translation site by site, with the Japanese beside it, and say whether the English really does what the Japanese does at each place. Makino uses doubled sound-words — ピヨン／＼ (piyon-piyon), ピシヤ／＼ (pisha-pisha), ジリ／＼ (jiri-jiri) — six times. My translating notes were confident about these: English has iterative, sound-symbolic verbs of its own, went hopping, slapping, go sizzling and curling back on itself, and I wrote that they "cost nothing."

The reader accepted one of the six. Its verdicts on the rest: "onomatopoeia dropped, plain verb only", "mimetic reduplication flattened into plain verb". Of the seven kinds of device I had declared in the story, this leaves exactly one that transfers whole and unchallenged — printing a line of dialogue and then, as its own separate sentence, who said it. That is a craft finding I did not go looking for, it contradicts my own notes, and it is one reader on one story, so it wants replicating on a second.

The checking, briefly

Before anything was judged, an independent model was asked to attack the design. It came back with thirteen objections, four of them fatal if ignored, and I accepted all thirteen — two of them changed the translations themselves before a single judgment was made. One of its objections I could not fix: it predicted that my "clumsy" English would read as specifically Japanese-flavoured clumsiness, which would muddy everything. I rebuilt the clumsy version to avoid that, tested it, and it failed anyway — an independent reader attributed 25 of my 35 clumsy rewrites to Japanese. On a story whose content is full of geisha and yukata, I do not think that is fixable, and I have said so rather than rebuilt a fourth time to please the instrument.

Every quality judgment here remains provisional: the model jury that scores translations has not yet passed its calibration test.

Cost, and what is next

This session cost $1.22. The day stands at $4.69 of the $5 daily budget, so tomorrow's first session has almost nothing to spend and should plan accordingly.

Next up is more actual translating — the first night of the Thousand and One Nights, whose frame tale I finished earlier this week.

Nothing needs your attention. The one oddity I have mentioned twice — an OpenRouter account total that creeps upward by itself when this project is making no calls — is unchanged, and since I no longer use that number for anything it costs the work nothing.