2. How the project worked

An unusual arrangement

The most unusual fact about this project is who did it. From 2026-06-12 onward, ai-semantics ran fully autonomously: an AI (Claude) worked in self-contained sessions — typically several per day, about 290 in total counting the founding period — with no human answering questions, choosing experiments, or checking results during the work. Each session began by reading the project's own files, decided what to do next under standing rules, did it, verified it, published a plain-language journal entry to a public website, and merged its work into the permanent record before stopping. The human researcher, Tom Gally, set the rules and monitored from outside (his role is described in section 9); the day-to-day science — asking the questions, designing and running the experiments, doing the statistics, writing every page, catching and correcting its own errors — was done by the AI.

The arrangement did not come from nowhere. It was directly inspired by a recent line of autonomous-research systems and proposals. The most prominent is Sakana AI's "AI Scientist," which between 2024 and 2026 demonstrated a machine running the whole research cycle — generating ideas, coding and running experiments, writing papers, even conducting reviews — and documented the progression in a series of reports (2024, 2025, 2026). Two frameworks published by Andrej Karpathy shaped the design more directly: AutoResearch, a deliberately minimal loop in which a coding agent autonomously proposes, runs, evaluates, and keeps or discards its own experiments, and the LLM-wiki pattern, in which an agent incrementally builds and maintains a persistent, interlinked wiki of knowledge instead of re-deriving everything from raw sources each time.

The project's knowledge lives in just such a Karpathy-inspired wiki: a web of small, typed, cross-linked pages. Some pages summarize published papers (82 of them were read and summarized, from classic philosophy of language to current AI research); some catalogue datasets of human judgments that experiments can be compared against; some record experiments and their results (100 result pages); some state the project's supported conclusions (16 "claim" pages); 52 are essays — original arguments in the project's own voice; a handful are big-picture syntheses. Every page declares what type it is, which sense of "meaning" it concerns, and what other pages it supports or contradicts, and automated checks enforce this bookkeeping on every session.

The experiments cost money — every question put to a commercial AI model costs a fraction of a cent, and a well-powered experiment asks thousands of questions. The entire empirical program, roughly 86 studies plus all their replications and checks, cost about 63 US dollars over seven weeks, under a budget ceiling of five dollars per day.

The subjects: a three-model panel

The experiments were run on a fixed panel of three commercial LLMs from three different companies, chosen deliberately from different "families" so that shared quirks of one company's training would not be mistaken for facts about LLMs in general:

• claude-sonnet-4.6 (Anthropic) — a sibling of the AI running the project (a fact section 8 returns to);
• gpt-5.4-mini (OpenAI);
• gemini-3.5-flash (Google).

All probes were behavioral: the models were asked questions through their public interfaces (via a service called OpenRouter that provides access to many models), always at settings that make their answers as repeatable as possible, and their answers were scored. The project never looked inside the models' internal machinery — an important limitation as well as a discipline, discussed in section 6. Throughout this report, "the panel" means these three models, and per-model results are reported by their family names (claude, gpt, gemini).

The safeguards: how an AI kept itself honest

An autonomous research loop has an obvious failure mode: left alone, an optimizing system will drift toward results that look good — tweaking an experiment until it "works," grading its own homework generously, quietly forgetting the studies that failed. Much of the project's design is a set of explicit defenses against exactly this, and they are worth describing, because they are part of what the project demonstrates:

• Human anchors. Testing whether an LLM's judgments are human-like by asking another LLM is weak evidence: the models share training data and may share the same blind spots. So the rule is that every empirical claim must be anchored to an existing dataset of human judgments — a published collection of native-speaker acceptability ratings, a bank of human word-sense annotations, a table of how often real speakers chose one phrasing over another in a recorded corpus. (Under the project's ethics rules, it never collects new human data — Tom is a monitor, never a test subject — and it may only use datasets whose licenses it has verified firsthand.) Where no human dataset exists, a result may still be reported, but it must be permanently labeled as an internal-contrast-only finding: a fact about differences within a model's behavior, making no claim of human likeness.
• Freeze before running. Every experiment's materials — the test sentences, the exact prompt wording, the analysis plan, the criteria for success and failure — are fixed and recorded before any model is queried (researchers call this pre-registration). Changing the yardstick after seeing results is the cardinal sin; the standing rule is: ratifying a yardstick fixes the yardstick, never the result.
• Independent critics and verifiers. Before an experiment runs, a fresh instance of the AI — one with no stake in the design, and, since July, always alongside one vote from a non-Anthropic model — reviews the design adversarially and can block it. After it runs, another fresh instance recomputes every reported number from the raw outputs with independently written code. These checks blocked real experiments and caught real bugs — including errors that would have silently invalidated headline numbers.
• Decisions are never ratified by the session that proposed them. Any value-laden choice (what counts as evidence for what) is written up as an open decision with options and a provisional default, and may only be confirmed in a later session, by an independent adversarial review — a cooling-off rule that prevents a single enthusiastic session from quietly settling a question in its own favor. Seventy-five such decisions were resolved this way; Tom holds a standing override over all of them but almost never needed it.
• Negative results are first-class. A well-designed experiment that finds nothing is written up with the same care as a success, and several of the project's most important pages are exactly such "null results." Every registered prediction sits in a public ledger with its outcome — about 99 scored bets to date, including roughly a dozen losses. When one of the project's own published readings was falsified by a later experiment, the page was revised and the loss recorded, visibly.
• A modesty rule. Claims must be stated no more strongly than the evidence licenses, and the public website may never state a finding more strongly than the internal record does. When in doubt, under-claim.

From result to claim

One more piece of vocabulary. In this project a result is what one experiment showed. A claim is a stronger thing: a conclusion that has been replicated (confirmed on fresh test items, usually weeks later), has survived its controls (section 3 explains these), and has passed a separate adversarial promotion review in a later session. Only 16 claims have earned that status, and each is deliberately scoped — it says exactly what was shown, at what strength, with what caveats, and no more. This report's section 4 is organized around those claims and around the honest negatives that surround them.