3. The central idea: the distributional shadow

To understand what the experiments found, one idea is essential. It is the project's workhorse, and it is a way of making the "stochastic parrot" worry precise enough to test.

Here is the worry in its simplest form. Suppose a model completes salt and ___ with pepper. Does it know something about the meanings of salt and pepper — their kinship as seasonings? Not necessarily: the two words simply occur together, constantly, in text. A system that tracked nothing but word-company — which words appear near which — would get this right for free. Call everything that such a system could do the distributional shadow of language: the image of meaning that is already cast by mere co-occurrence statistics, the patterns of which words keep company with which. The shadow is not nothing — it is genuinely informative, which is exactly why prediction-trained models learn so much from it. But behavior that stays inside the shadow gives no evidence of anything beyond pattern-matching over word company.

So the decisive question for any meaning-like behavior is: does it go beyond what the shadow predicts? And that question can be answered experimentally, by building a control — a comparison condition that captures the relevant statistics while removing the meaning. Three kinds of control recur throughout section 4:

• Same-words controls. Test the model on sentences that use the very same words but lack the meaningful pattern. If the behavior tracks the pattern and not the words, it survives; if it was really keyed to the words, it collapses.
• Statistical partialling. Measure the distributional quantity directly — word frequency, or how alike two sentences' vocabulary is — and subtract its contribution statistically, asking whether any effect remains (the residual).
• Firewalls. The strictest form: design the experiment so the two things being compared are byte-identical — literally the same string of characters — with only the surrounding context differing, or use nonce words (invented words like the wug's blanket) that have no statistics to exploit. Any effect that survives such a control cannot be a fact about the tested string's word statistics, because there is literally nothing statistical to distinguish.

A behavior that survives its control is called, in the project's shorthand, a shadow-beater: it shows the model tracking something over and above word-company statistics. A behavior fully explained by its control is shadow-saturated. And one of the project's central discoveries — elaborated in section 5 — is that phenomena differ enormously in shadow-depth: some meaningful phenomena leave such a heavy statistical footprint in text that surface patterns alone reconstruct them (so a model's success there tells you little), while others leave almost no footprint, so that success there is informative. Mapping which is which — phenomenon by phenomenon, with measured residuals and error bars — became the project's flagship deliverable: the shadow-depth table.

One caution before the tour. "Beats the distributional shadow" does not mean "possesses true understanding." It means: this behavior is not explained by the specific statistical shortcut this control captures. There may always be a subtler shortcut no one has thought to control for — an open worry the project takes seriously (section 6 discusses the deepest known one, construction frequency). The claims are deliberately built to be exactly as strong as their controls, and no stronger.