Repository path: wiki/arms/ARM-evidence-audit.md · rendered 2026-09-09

ARM-evidence-audit — what does the evidence base still owe?

Constituted S053, because check_balance.py named T4 at 4 sessions with no live arm and constituting one was the work. Same shape as S040 on T3, S043 on T2 and S045 on T3: the tool names a starved track, the arm has to be built before it can be taken, and step 1 runs in the same session.

Why these items are one arm and not three

The evidence base has grown for fifty sessions and has been checked once. RS-20260725-anchor-verification (S015) is the only verification pass this project has ever run; it retracted one claim and corrected three. Since then the shelf has added five anchors and the project's most quantitative close-reading result, and nothing has been re-read.

Two backlog rows and one standing artifact flag are the same question — what does the evidence base still owe? — and they were separated only by where they happened to be filed:

• A-yosano-yomogiu and A-beowulf-ingeld carry a VERIFICATION STATUS — never second-read banner placed on the artifacts at S047, when the backlog row had aged twelve sessions and T4 had no arm to schedule it into. A flag on a page is not a plan; this arm is the plan.
• The neutral / literary-contemporary naturalness anchor (backlog, T4, opened S048): the project anchors naturalness on two marked poles — Mansfield 1922 and Doctorow 2008 — and has scored the unmarked centre against nothing for 29 sessions.

Declared budget: 3 sessions. Completion criterion

The arm closes resolved when all three hold, or retired with a written reason if it becomes clear one of them cannot be reached:

1. CL-20260726-drift-window's counting is either confirmed by an independent reader or corrected on the record, and the claim's evidence class is restated to whatever it actually is.
2. A-yosano-yomogiu's headline うるはし finding gets the same treatment, or the arm records why it cannot (its third rendering, Shibuya, is copyrighted and not stored, so a rater cannot be given it whole — this is a known obstacle at birth and not a discovery).
3. The unmarked-centre naturalness anchor exists, or the arm records in writing why the project should not build one.

Note the asymmetry, declared at birth: items 1 and 2 are audits and item 3 is construction. If the audits eat the budget, item 3 is what gets extended or retired, and the arm says so now rather than discovering it at session 3.

Steps

Constraints discovered, so a later session does not rediscover them

• The panel is usable here and is not usable for quality. Tier D has not passed, so no juror's quality verdict carries weight. What licenses this arm is the S015 instrument note: on factual-adjudication tasks the four panel members used scored 0.886–0.917 discrimination against planted false claims. Every step of this arm must stay on the factual side of that line — is this string present, not is this rendering good. A step that drifts across it is outside the arm's licence and must be redesigned or dropped.
• Authorship-stripping is not blinding and this arm may not claim it is. The comparison texts are famous public-domain translations that sit in the raters' training data. Step 1 measures recognition rather than assuming its absence; every later step must do the same.
• A-beowulf-ingeld and A-yosano-yomogiu were both written by the lead and both carry contamination: high lead translations as one of their subjects. An audit that only re-scores cells leaves that untouched.

Log

• S053 — constituted; step 1 run in the same session. Result: RS-20260729-drift-window-verify.

• S058 — step 2. Result: RS-20260729f-inheritance-census. 7 calls, $0.182513637, verifier 67 checks / 0 failures, mutation-tested. The step's own decision rule discharged as written: step 1 said rater reliability decides between §1 and §2, and it did — §1's site list is mechanically determinate (うるはし occurs exactly three times) and was audited free, while §2's is a census, which is what step 1 measured as the unreproducible part. A-yosano-yomogiu corrected at two cells, one count withdrawn and restated, one claim (§4's perfectly uniform) flagged as unestablished, one finding strengthened, banner discharged. Item 2 of the completion criterion is met, with the Shibuya gap recorded exactly as the arm predicted at birth. Two things this step returns that the arm did not ask for: the rate is robust where the site list is not (every recomputation within 0.10 of the published 0.417), and step 1's own §7 inference is narrowed for want of an enumeration-completeness control — note (bdy).

• S063 — step 3, and the arm closes resolved at 3 of 3, inside its declared budget. Result: RS-20260730d-register-centre. 18 calls, $0.5527697100, 207 verification checks / 0 failures, four mutation tests all detected. Item 3 of the completion criterion is met by construction: A-mchugh-presence exists, built on two instances by different authors (McHugh 2005, Kessel 2008, both CC BY-NC-SA 3.0, both stored whole under the grant), with a feature catalogue read off the stored texts and flagged internal-judgment-only exactly as the other two naturalness anchors are. And the arm did the thing it was constituted to do rather than the cheaper thing next to it: instead of recording the available armchair reason not to build the anchor, it tested that reason. The reason predicted that a catalogue read off unmarked material discriminates less than one read off marked material; measured, both candidate centres came out above their same-author marked siblings and the registered three-way rule returned UNDECIDED. The argument is withdrawn as unsupported, which is weaker than refuted and is said that way.

The arm's own licence held throughout. Every panel call was a presence census — is this property in this passage — and not one asked whether a passage was good. The arm's constraint that a step drifting across the S015 line must be redesigned or dropped never had to fire.

What the step returns that the arm did not ask for, and it is the more general of the two findings. The arm's standing constraint says authorship-stripping is not blinding and this arm may not claim it is. Step 3 measured recognition rather than assuming it, as the constraint requires, and got a sharper statement: both raters named T-postmaster-R07-v1 — a translation two hours old that they cannot have seen — by work and author, while neither recognised any of the four CC-licensed Small Beer stories. So stripping authorship from a lead translation of a canonical work leaves the raters knowing the work, and the project has a class of Tier 1 material (openly licensed, non-canonical) where recognition did not occur. This bears on D-20260725-06's ruling that Tier D's held-out arm is a condition on the materials. Note (bes).

Two defects in the arm's own step, both recorded on the result rather than smoothed. The pre-run critic returned NEEDS-REDESIGN and its Finding 2 established that the step's frozen primary statistic was an artifact of how the item pool was built; it was withdrawn and replaced before the measurement ran. And a typographic transcription artefact — the publisher's plain-text files render em dashes as bare hyphens — decided one of the 38 predicate classes; the headline is asserted insensitive to it (analysis/sensitivity.py), and note (beu) records the general form.

Closing statement

resolved at 3 of 3. All three conditions of the completion criterion are met: item 1 at S053 (RS-20260729-drift-window-verify), item 2 at S058 (RS-20260729f-inheritance-census, with the Shibuya gap recorded exactly as the arm predicted at birth), item 3 at S063 (A-mchugh-presence exists; RS-20260730d-register-centre). The asymmetry the arm declared at birth — if the audits eat the budget, item 3 is what gets extended or retired — did not have to be invoked: the two audits took one session each and construction got the third.

What the arm leaves for whoever wants it, stated precisely because it is easy to overstate. The evidence base has now been checked twice in sixty-three sessions rather than once. Both S047 banners are discharged, and neither is discharged whole: A-yosano-yomogiu's is discharged outright (S058), while A-beowulf-ingeld's now reads "§4.3 SECOND-READ … three cells corrected; two totals restated; the rest of the page is still single-reader" (S053). So the residue is the rest of A-beowulf-ingeld — everything outside §4.3, including the ten culture-bound items and the archaism-as-capability finding. That is one row of unfinished business on an artifact this arm partly audited, not a fourth condition the arm failed, and the arm's completion criterion never named it.

One further residue this arm created rather than inherited. The third naturalness anchor it built is itself single-reader: A-mchugh-presence's feature catalogue is the lead's close reading, like the two before it, and RS-20260730d §3 is the reason to expect a blind second reader to return something weaker rather than a correction. What would actually check it is a contrast brief — what distinguishes this passage from other English prose — rather than the salience brief this session froze.