Repository path: workshop/experiments/E-20260803e-purpose-index/design.md · rendered 2026-09-09

E-20260803e — is "natural" one judgment or two? Corpus-indexed against purpose-indexed markedness

ARM-fluency-record step 3 of 3 (T4). Frozen before any dispatch.

0. The subject-rule sentence (continue-prompt.md §4.5), written before the unit was designed

What does this unit teach about translating literature or evaluating translations? — It teaches whether a reader's sense of "this English reads naturally" is one judgment or two: whether the same sentence can be at once inside the repertoire a reader brings to the kind of writing it is, and marked against present-day standard English. If it can, then a translator working in a genre-marked register is not failing naturalness but satisfying a different index of it, and an evaluation that scores such a translation against a period corpus is scoring the wrong thing.

That sentence is about readers and translations. The consequence for this project's sense list follows from the finding; it is not the question.

1. What is asked, and why this arm owes it

wiki/goodness-senses.md, naturalness, since S097: Arnold's 1861 established-possession test — "whether a diction is antiquated for that particular purpose for which it is employed" (S-arnold-newman-homer §2) — is recorded as a candidate refinement and NOT adopted, because "adopting it needs a measurement ARM-fluency-record is constituted to make and has not made." This is that measurement.

naturalness currently indexes markedness to three period corpora. Arnold's test indexes it to purpose and genre. The empirical question on which adoption turns is whether those two indices are actually different — whether they can be brought to disagree, and in which directions.

2. Materials

The translation limb, frozen and committed before this design was written. A matched R10 pair on 楠山正雄「かちかち山」 (Aozora 000329, PD, 4,177 non-whitespace characters, complete):

• T-kachikachiyama-R10t-v1 — target A-english-tale-register (purpose-indexed), 2,134 words.
• T-kachikachiyama-R10u-v1 — target A-mchugh-presence (corpus-indexed, unmarked/literary- contemporary), 1,734 words.

Same source, same translator, same session; only the target catalogue varies (R10). Contamination none, measured on a pre-selection gate before either translation began: 0 shared 7-grams, longest common run 4 tokens against Ozaki 1908, the freely reachable published English rendering of this tale, which was extracted to disk and never read.

The item set, materials/items.json, 34 items (amended, §4a), six classes:

Twenty-two of the thirty-four items are verbatim from a translation frozen before this design existed, and that is checked mechanically by analysis/verify.py. The twelve constructed items are declared as constructed here and on the result page. Class X is Arnold's own negative list and its kin: the words he says are not an established possession — and the anchor corpus, which is saturated with quoth (22) and whereupon (6), contains none of them.

The marked element of each item is delimited by «guillemets». Every item is 1–3 sentences.

3. The manipulation — one text, two questions

Nothing about the material changes between conditions. Only the question does. Each seat sees the same 34 items twice, in two independent API calls with no shared context (so there is no carryover by construction). Both prompts open with the identical material description — "These passages are all from an English translation of a traditional Japanese folk tale — the kind of story that is handed down and read aloud" — so the genre frame is not what varies (amendment A1). What varies is:

• CORPUS index. "Against present-day unmarked literary English prose — the ordinary register of contemporary literary fiction, whatever the writing is for — how MARKED is the delimited element? 0 = wholly unmarked, 4 = maximally marked."
• PURPOSE index. "Matthew Arnold, writing in 1861 …, proposed this test: the question is 'whether a diction is antiquated for that particular purpose for which it is employed'. Apply that test to the purpose these passages are written for. Is the delimited element within the established possession of a present-day English reader FOR THAT PURPOSE? 0 = wholly within, 4 = wholly outside."

Both scales run in the same direction, so a per-item gap = CORPUS − PURPOSE is meaningful. The purpose prompt's illustrative sentence was removed by amendment A1: it said a feature could be old and still be an established possession, which is a licence to downscore archaism present in one condition and absent from the other, and P1 could have been produced by it alone. Arnold's bare question now does the work.

Every item is truthfully described in both conditions. Every excerpt is from a translation of a traditional folk tale. No seat is told anything false.

4. Seats and dispatch

Jurors P1 openai/gpt-5.6-terra, P3 x-ai/grok-4.5, P5 deepseek/deepseek-v4-pro — the three non-Anthropic seats used as coders at S097, and here again coders of a descriptive property, not judges of quality. No translation is evaluated for quality anywhere in this run; Tier D is NOT PASSED and nothing here needs it.

3 seats × 2 indices × 2 item orders (registered forward order and its exact reverse) = 12 calls, 34 items each. The order arm is a robustness control, not a replication for power. Each call is stateless. The frozen order is materials/order.json, generated before dispatch.

Pre-run critic: P4 moonshotai/kimi-k3, which grades nothing in this experiment, with reasoning: {"effort": "low"} on the first dispatch — note (b)'s standing amendment, the one S097 failed to apply on this exact seat at a cost of $0.260301.

4a. AMENDED after pre-run critic pass 1 — read §10 before §5

Pass 1 returned NEEDS-AMENDMENT, seven findings, four BLOCKING. All seven were accepted and are applied below and in §10. Nothing had been dispatched. The item set went from 30 items to 34 (a new class M2), two prompt sentences changed, four item spans were re-cut, three neutral controls and both attention items were replaced, and two criteria were repaired. §5 as it now stands is the amended version and is what was frozen for dispatch.

5. Registered predictions and criteria

Let gap(item) = mean CORPUS − mean PURPOSE over the 6 (seat × order) cells.

PRIMARY — the purpose index is a DISTINCT INDEX, not a leniency adjustment toward old words, iff all FOUR of these hold (P4 added by amendment A2):

• P1 — the expected direction. Class P mean gap ≥ +1.00. (Repertoire features are less marked when judged for the purpose.)
• P2 — THE REVERSAL, and the discriminating test. Class M mean gap ≤ −1.00 — i.e. contemporary-standard features are judged more marked under the purpose index than under the corpus index. A single scale cannot produce this; only a second index can.
• P3 — discrimination WITHIN archaism. Class X mean |gap| ≤ 0.50, with class X CORPUS mean ≥ 2.50 AND class X PURPOSE mean ≥ 2.50. (Old words that are not in the repertoire stay marked under both indices.) Amended A6a: as first written, |gap| ≤ 0.50 could be satisfied by both indices scoring X low — a seat that knows anon from Shakespeare and rates it 1 twice would have passed a criterion named "these words are not possessed". Both means must now be high.
• P4 — context independence (new, amendment A2, critic finding 2). Class M2 mean gap ≤ −0.50 — the same four contemporary features as four of the M items, constructed into tale-register sentences. If M's reversal is driven by a contemporary feature clashing with its own contemporary surroundings, M2 should not reverse. P4 is required for the "distinct index" branch of the map; without it the run cannot separate outside the reader's repertoire for this purpose from incongruent with the register of the sentence it sits in.

CONTROLS — gates on interpretability. If any fails, the primary is withheld and reported as uninterpretable.

• F1 — no global scale shift. Class N mean |gap| ≤ 0.50. Amended A3: the three neutral items the critic identified as oral-formulaic — thump, thump, one, two, three, and a tripled-adjective span — are removed. Verbatim agreement between two independently-targeted renderings anchors provenance, not genre-neutrality, and two renderings agreeing on onomatopoeia does not make onomatopoeia neutral. All six N items are now plain narrative description carrying no oral-formulaic or genre marking, still verbatim-shared.
• F2 — attention. Both class A items score ≥ 3.0 under both indices in ≥ 5 of 6 (seat × order) cells. Amended A5: the attention items were scrambled word order, and the same prompt teaches the seat that inversion (Up jumped the old man, said he) is repertoire — so a diligent purpose-index seat could read the scramble as extreme archaic inversion and score it low. The failure would have been asymmetric, more likely in exactly the runs where the seat took the purpose frame most seriously. Both items are now ungrammatical by agreement and morphology (the hare have took … and cutted, his legs was gave way under of him), which no register of English licenses.
• F3 — completeness. Every call returns 34 parseable ratings. A body with fewer is a seat failure in the runner, whatever finish_reason says.
• F4 — order robustness. The sign of each class-level gap (P, M, M2, X, N) is the same in the forward and reversed orders, for any class whose forward |gap| ≥ 0.50. Amended A6b: as first written, a class gap of +0.2 flipping to −0.1 — noise either side of zero — would have failed F4 and withheld the primary.

Registered secondary, reported as context and NOT gating: per-item hit rate — the proportion of class-P and class-M items whose individual gap has the predicted sign.

6. Result → option map, fixed before dispatch

In every branch the arm closes at 3 of its 3 declared sessions. It does not extend. This is recorded now so that no outcome can be read as licensing another Homer step (ARM-fluency-record, "Do not extend the arm to translate more Homer").

7. What this design cannot show, registered as limits before the numbers exist

1. One genre, one direction. The purpose tested is traditional tale. Whether purpose-indexing behaves the same for epic, for legal prose, for a modern realist novel, is untested. Arnold's own case was epic verse and this run does not reach it.
2. The raters are language models, and the question is about a reader's repertoire. They are used here as coders of a descriptive property, which RS-20260725-anchor-verification supports for factual/linguistic coding, and not as a reception record. No claim about human readers follows.
3. Class X is constructed and its items are rare words. A seat may score them high because they are unfamiliar rather than because they are unpossessed-for-the-purpose. The design cannot separate those, and P3 is therefore a weak discriminator in the passing direction and a strong one in the failing direction.
4. The lead wrote both renderings and assigned every class. Class assignment for P and M is anchored by mechanical attestation counts; class N is anchored by verbatim agreement between two independently-targeted renderings; class X and A are not anchored by anything but the lead's construction.
5. A tale register is still a target-culture norm. Nothing here bears on whether rendering 昔話 into English nursery idiom is a good idea; A-english-tale-register says so in its own text.
6. Authority cue and prompt asymmetry, residual after A1 (critic finding 7a). The purpose prompt still names Arnold and is still the longer of the two. Naming an authority shifts ratings on its own. A1 removed the substantive asymmetry — the illustrative licence and the differential genre frame — and this residual is declared rather than removed, because the test cannot be stated without stating whose test it is.
7. Within-call anchoring, uncontrolled (critic finding 7b). Every call contains all 34 items, so the presence of eftsoons in the list recalibrates what whereupon looks like. Class composition is constant across the order arm, so the order control does not touch this. Every number here is relative to the item pool it was rated in.
8. The memorisation pathway, and the demand pathway beside it (pass 1 finding 7c; pass 2 finding 1). A seat asked whether something is in a present-day reader's repertoire has no reader's repertoire; it may instead be pattern-matching to 19th-century tale collections in its training data — of which Jacobs 1890 is certainly one. On that reading P1 measures familiarity with the Jacobs-flavoured corpus, not an index. The design cannot exclude it, and P4 and the M2 class are the only parts of the run that bear on it at all. Pass 2 sharpened this and the sharpening is accepted: M2 controls the local register-clash route but not the demand route — a seat told "traditional tale" and asked about a reader's repertoire may score contemporary phrases as outside it wherever they sit. So a P1–P4 pass licenses "the two indices behave differently when a rater is asked to apply them", and not "English readers possess this repertoire".
9. The shared genre frame can only attenuate P1, never create it (pass 2 finding 4, registered so a marginal result cannot be re-litigated after the fact). Both prompts now name the folk-tale purpose, so some frame will bleed into the CORPUS condition. Bleed makes the corpus condition more lenient to archaism, which lowers class-P corpus scores and compresses the P1 gap. For class M there is no archaism to be lenient toward, so P2 is roughly untouched. A P1 pass is therefore harder to obtain than the design intends, and a P1 near-miss is a weaker result than it looks. This is the trade amendment A1 deliberately made: a prompt asymmetry that could manufacture P1 was exchanged for a frame bleed that suppresses it.

10. Amendment record — pre-run critic pass 1

Seat P4 moonshotai/kimi-k3, effort: low on the first dispatch, $0.073764, finish_reason: stop. Verdict NEEDS-AMENDMENT, seven findings, four BLOCKING, ALL SEVEN ACCEPTED. Note (rr), forty-first consecutive session with an independent pre-run critic pass.

Nothing was dispatched before these amendments. The amended design, item set and both assembled prompts went to a second critic pass before the scoring stage.

8. Verification

analysis/verify.py recomputes every reported number from the stored raw JSON bodies with a parser independent of the scorer, re-checks all 22 verbatim item provenances against the frozen translations, re-checks all attestation counts against the stored anchor corpora, re-sums the billed cost, and runs mutation tests each asserting that the bytes on disk changed and each restoring the mutated file (notes (bgu), (bhd)).

9. Budget

Declared worst case built from max_tokens (note (abc)), not from expected output:

Against $2.84 of headroom remaining on UTC 2026-08-03 after this session's ratification gate ($0.04527425). A run that does not fit is split or deferred.

11. Amendment record — pre-run critic pass 2

Same seat (P4, effort: low), $0.102372, finish_reason: stop. Verdict OK-TO-RUN. It was shown pass 1's report and asked to check closure rather than re-report. All four of pass 1's BLOCKING findings CLOSED, checked against the assembled prompts and item strings rather than against the design's claims about them. Four new NON-BLOCKING findings, all four accepted:

Two critic passes, eleven findings, all eleven accepted. Note (rr) holds. Combined critic cost $0.176136 against the declared $0.36.