2026-10-25 (second report): reviewer A becomes a Claude model
Interactive session 20261025T014638Z-ysf43k, at the owner's request. Mode: setup (an experiment, then a change of reviewer).
What was done
The owner asked whether the two paid reviewers could be replaced by Claude models running inside the owner's Claude subscription. The reviewers are the two outside AI models that check every field of every new entry; they are reached through the OpenRouter service and paid per call.
First, an experiment (experiments/claude-reviewers-v1/README.md). Twenty-eight entries were reviewed by six reviewer setups, all using exactly the review instructions the dictionary already uses:
- Twenty were finished entries with fifty errors deliberately planted: thirty plain ones and twenty subtle ones, such as promoted to a manager or the noun's stress on the verb record.
- Eight were fresh drafts of words still waiting in the queue. They were kept in the experiment folder only.
Each problem a reviewer raised was then judged blind: all the issues for an entry were pooled and shuffled, and the reviewers' names were hidden. A Claude judge decided whether each was right. Because a Claude judge might favour Claude reviewers, the two outside models judged the same pools as a check.
Then, the change the owner chose. Reviewer A, previously an OpenAI model, is now a mid-size Claude model. It runs as a "headless" call: a single command-line request to Claude Code, with no tools and only the review instructions, inside the owner's subscription. Reviewer B, the Google model, is unchanged, so the two reviewers still come from different companies. The tools that call reviewers, record their decisions, and measure their accuracy were updated, and new tests were added. The decision is recorded in wiki/decisions/reviewer-a-claude.md.
What was found
- The new reviewer is better. The old reviewer A was right on 65–87% of its "blocking" calls, depending on which judge decided; blocking means a fix is required before the entry can be published. It raised about one false alarm per entry and missed five of the twenty subtle errors. The Claude reviewer was right on 97–99% of its blocking calls and found all fifty planted errors.
- The new pair is the strongest panel. Paired with reviewer B, the Claude reviewer found 63 of the 71 real blocking problems, with only 2 false alarms. The old pair found 62, with 33 false alarms, and each false alarm costs the session time to read and reject.
- How the Claude reviewer is called matters. Started as a "subagent" (a helper session), it used about thirteen times more tokens than the headless call, because each subagent loads the whole Claude Code instructions. At the default effort setting it hardly reasoned and got less accurate, so it now runs at high effort.
- A failure found while building: asking the command-line tool for "any JSON object" returned an empty verdict list. It now receives the exact shape of a review reply.
- The experiment cost US$3.60 on OpenRouter. This is now recorded in the budget ledger. Each review by the new reviewer A costs nothing in money; it uses about 12,000 to 15,000 subscription tokens and takes about 20 to 30 seconds.
What it means
Reviews should get more accurate, with fewer false alarms to reject, and OpenRouter spending on reviews should fall by roughly a third. A batch of entries takes about as long as before, because the two reviewers now run side by side.
What is next
- The next lint pass (the periodic quality audit) should judge the new reviewer A by its own figures, which
tools/metrics.py --precisionnow reports separately. - Four issue types that had been switched off for the old reviewer are on again for the new one, and will be measured afresh.
What needs the owner
PROJECT.mdstill says the reviewers are "two non-Anthropic reviewers". Sessions do not edit the charter, so that line is yours to update.- Your 2026-10-05 ruling switched off reviewer A's pronunciation checks because the old model was rarely right about them. They stay off until you say; the new model caught all four planted pronunciation errors.
- Seven branches can be deleted; they are listed in
NEXT.md. This session could not delete branches itself.