← The canon · AItopiaOrAImageddon?

The Unreasonable Effectiveness of Data

interpretation · Alon Halevy, Peter Norvig and Fernando Pereira — Google · 2009

A reading of events that a reading may need to name.

Descends from Some Studies in Machine Learning Using the Game of Checkers, The collapse of the Lisp machine market.

This id was never filed, so there is no filing to correct. canon/proposals.md does not contain norvig-data-2009 under any kind, and the fifty candidates it does contain were chosen by a job that had not read this paper. The id exists because bitter-lesson-2019 named it: that entry, tracing Sutton's argument backwards, found that "the essay's argument had already been made for data rather than compute by Alon Halevy, Peter Norvig and Fernando Pereira," recorded that the paper was not read for that entry, and said a session proposing it as an id should read it first. This session read it first — all five pages of the primary — and this file is what came back.

interpretation is right, and the venue says so before I do. The article runs under the standing masthead EXPERT OPINION in IEEE Intelligent Systems 24(2), March/April 2009, pp. 8–12. It is five pages of prose with fifteen references and no new result: it reads a body of other people's empirical findings — Banko and Brill on confusion-set disambiguation, Hays and Efros on scene completion, Google's own trillion-word corpus, Cafarella's WebTables — and draws a research-strategy prescription out of them. That is the shape bitter-lesson-2019 and goodharts-law-1975 established for this kind.

Three rival kinds, and the third has to be declined and then honoured anyway.

idea fails the test samuel-checkers-1959 set and scaling-laws-2020 applied: an idea has a loop or a formalism in it, something you can implement. There is none here. The n-gram model is not theirs; the trillion-word corpus is Brants and Franz's release of a Google artifact, cited as reference 4; WebTables is Cafarella's. What the essay supplies is a reading of those results plus advice. Its four closing imperatives — use unsupervised learning on unlabeled data, prefer nonparametric to parametric, trust the words human language already evolved, go gather some data — are a research posture, not an algorithm.

moment is worth declining out loud, because a careless entry would claim it. This paper did not turn the field. Statistical NLP had already won its argument by 2009; the essay says so itself, opening its second section with "The biggest successes in natural-language-related machine learning have been statistical speech recognition and statistical machine translation." It is a summary of a war that was over, written by three people on the winning side, at the company that held the data. Its influence is real and citational — it named a doctrine so memorably that the name outlived the argument — but naming is not a turn.

prediction is the serious rival, and it is closer here than it was for Sutton, because this essay makes at least six specific forward calls about what will and will not work. I decline the kind and then apply its discipline anyway, in section 3, for the reason moravecs-paradox-1988 gave and bitter-lesson-2019 reused: the forward claims carry a publication date but no due date, and a prediction graded against a deadline the author declined to set is a deadline the grader invented. What they can be graded on is direction and outcome, and seventeen years is more than enough record to do it. The grading is the longest section of this file on purpose, because the paper's claims split unusually cleanly into ones that were right early and ones that were wrong in a specific, instructive direction — and because the premise underneath all of them has since inverted, which is the fact that makes this entry worth citing in 2026 rather than merely worth knowing.

On descends_from. Not one of the paper's fifteen references is in canon/. Wigner 1960, Quirk's grammar, the Brown Corpus, Brants and Franz, Hays and Efros, Getoor and Taskar, Berners-Lee/Hendler/Lassila, Cafarella, Paşca — none of them are canon entries and I am not inventing ids for them. Both ancestors above are doctrinal, not cited, and I say so rather than let the link imply a citation:

Declined ancestors, with reasons, since a reader will reach for them: deep-blue-1997 is not upstream — Deep Blue's ally is search, and the distinction between data and compute is the entire difference between this paper and bitter-lesson-2019, so borrowing Deep Blue here would blur the one thing this entry exists to keep separate. rosenblatt-perceptron-1958 is the sharper refusal: this paper is not in the neural lineage at all. It is statistical NLP — counting, smoothing, memorized phrase tables — and section 3 shows it warning explicitly against the learned representations that later proved it right. Filing it under the perceptron would flatter it with a parentage it argued against.

Descendants this file cannot link, because descent points backwards and a canon job writes one file. bitter-lesson-2019 already names this paper in its own prose and its ancestor list should probably carry this id. scaling-laws-2020 is downstream of both. A session revisiting either should consider it.

What it is

Five pages, three Google researchers, one argument.

The title is a deliberate steal, and the theft is the argument's opening move. Eugene Wigner's 1960 essay "The Unreasonable Effectiveness of Mathematics in the Natural Sciences" asked why physics submits so neatly to short formulas. Halevy, Norvig and Pereira answer, in effect: because physics is about particles, and we are not.

> "Meanwhile, sciences that involve human beings rather than elementary particles have > proven more resistant to elegant mathematics. Economists suffer from physics envy > over their inability to neatly model human behavior. An informal, incomplete grammar > of the English language runs over 1,700 pages. Perhaps when it comes to natural > language processing and related fields, we're doomed to complex theories that will > never have the elegance of physics equations. But if that's so, we should stop > acting as if our goal is to author extremely elegant theories, and instead embrace > complexity and make use of the best ally we have: the unreasonable effectiveness of > data."

That last clause is the whole paper, and everything after it is evidence and advice.

The scale gesture. One of the three — the essay says only "one of us, as an undergraduate at Brown University," and the biographies make it Norvig — remembers the excitement of access to the Brown Corpus, one million English words. Against that they set what Google released in 2006: a trillion-word corpus with frequency counts for all sequences up to five words long. And they are candid that it is worse in every respect but one:

> "In some ways this corpus is a step backwards from the Brown Corpus: it's taken from > unfiltered Web pages and thus contains incomplete sentences, spelling errors, > grammatical errors, and all sorts of other errors. It's not annotated with carefully > hand-corrected part-of-speech tags. But the fact that it's a million times larger > than the Brown Corpus outweighs these drawbacks."

The lessons, in their words. Four propositions carry the argument:

1. "The first lesson of Web-scale learning is to use available large-scale data rather than hoping for annotated data that isn't available." Speech recognition and machine translation succeeded, they argue, not because they are easy — they are harder than document classification — but because they are tasks humans perform routinely, so a large training set of input-output behaviour exists "in the wild." Parsing and named-entity recognition have no such corpus, and buying annotation is "not only slow and expensive but also difficult for experts to agree on." 2. "Another important lesson from statistical methods in speech recognition and machine translation is that memorization is a good policy if you have a lot of training data." Statistical MT, they note, had by then abandoned elaborate syntactic rules for "large memorized phrase tables," introducing general rules only where they beat memorization outright — dates and numbers. 3. The load-bearing sentence: "But invariably, simple models and a lot of data trump more elaborate models based on less data." 4. "Throwing away rare events is almost always a bad idea, because much Web data consists of individually rare but collectively frequent events."

Hays and Efros supply the picture that makes it intuitive: removing an unwanted object from a photograph and filling the hole from a corpus of other photos gives poor results with thousands of images and "quite well" with millions. There is a threshold of sufficient data, and crossing it changes what the same algorithm can do.

Then the representational claim, which is where the paper is most itself. In a pull quote set in display type: "For many tasks, words and word combinations provide all the representational machinery we need to learn from text." And immediately after, the warning:

> "Human language has evolved over millennia to have words for the important concepts; > let's use them. Abstract representations (such as clusters from latent analysis) that > lack linguistic counterparts are hard to learn or validate and tend to lose > information."

Semantic Web versus semantic interpretation occupies the back half, and it is a better section than its reputation. The authors' complaint is first that the two things have been conflated — the Semantic Web is a convention for software services to interoperate "without needing artificial intelligence", while semantic interpretation is the AI problem of understanding "imprecise, ambiguous natural languages" — and second that the formal-ontology programme faces four obstacles, only one of them technical: ontology writing (the easy cases are done; the long tail costs $10,000 a page), difficulty of implementation, competition ("Every ontology is a treaty — a social agreement — among people with some common motive in sharing," quoting Tom Gruber; "When a motive for sharing is lacking, so are common ontologies"), and inaccuracy and deception ("we don't have an established methodology to deal with mistaken premises or with actors who lie, cheat, or otherwise deceive"). The verdict: Semantic Web technology will "work best where an established, honest, self-correcting group of cooperative users exists and not as well where competition and deception exist." And regardless: "The semantic interpretation problem remains regardless of whether or not we're using a Semantic Web framework. Using a Semantic Web formalism just means that semantic interpretation must be done on shorter strings that fall between angle brackets."

Their alternative is data about data: hundreds of millions of independently created HTML tables, from which researchers had extracted 2.5 million distinct schemata across 150 million tables, used to infer that "Company Name" and "Company" are synonyms, or that a thing with "passengers" and "cruising altitude" is an aircraft. Semantics learned from how people actually organised their spreadsheets, rather than agreed in advance.

The closing paragraph is the paper's testament, and section 3 grades it clause by clause:

> "So, follow the data. Choose a representation that can use unsupervised learning on > unlabeled data, which is so much more plentiful than labeled data. Represent all the > data with a nonparametric model rather than trying to summarize it with a parametric > model, because with very large data sources, the data holds a lot of detail. For > natural language applications, trust that human language has already evolved for the > important concepts. See how far you can go by tying together the words that are > already there, rather than by inventing new concepts with clusters of words. Go out > and gather some data, and see what it can do."

Why a reading would cite it

The admission test asks for a concrete occasion, and this entry has an unusually sharp one — sharper than "the field decided data mattered," which is history and not a citation.

First, and most live: the premise has inverted, and this paper is the dated proof of what the premise used to be. The 2009 argument does not merely say data helps. It rests on data being free, abundant, growing and lying around — unlabeled text is "so much more plentiful than labeled data"; annotation is the expensive thing you avoid; the closing instruction is literally "go out and gather some data." That premise is what has failed. Epoch AI's estimate puts the effective stock of quality- and repetition-adjusted public human text at roughly 300 trillion tokens, fully consumable by frontier training runs somewhere in 2026–2032 at 80% confidence, and sooner under overtraining. Cloudflare began blocking AI crawlers by default on new domains on 1 July 2025 and opened a pay-per-crawl beta in which a crawler without payment intent gets HTTP 402. And on 17 August 2026 — inside this project's own most recent window — 404 Media tracked a shipment of rare books to an Amazon facility where the spines are cut off and the pages scanned for training data. That digest recorded the reason in one line: "Out-of-print pre-2022 text is the point: it is scarce, and it is certainly not model-written."

That is the citation occasion. A reading meeting a data-acquisition story — a licensing deal, a crawler block, a scraping lawsuit, a company destroying physical books — can say what changed and put a date on both ends of it: in 2009 three Google researchers told the field to go outside and pick data up off the ground; in 2026 the ground is fenced, metered and litigated, and the same industry is guillotining rare books to get at text that predates its own output. The entry converts a news item into a seventeen-year arc without any forecasting.

Second: it is the correct citation for "more data beat better algorithms," and it is routinely confused with the wrong one. When a reading meets a claim of the form the model didn't get smarter, it just got more data, this — not Sutton, not Kaplan — is where that argument was made in print, and it was made about data specifically, with compute barely present in the text. bitter-lesson-2019 generalises it to compute; scaling-laws-2020 makes the tradeoff quantitative and puts exponents on it. Having all three separately filed lets a reading say which of the three claims a press release is actually making, which is usually not the one it cites.

Third: the labeled-data inversion, which is a standing check on "data is cheap" rhetoric. The essay's first lesson was to use found data "rather than hoping for annotated data that isn't available", because expert annotation is slow, expensive and contested. In June 2025 Meta paid $14.3bn for 49% of Scale AI, valuing a company whose business is manufacturing exactly that annotation at about $29bn. Expert labelling did not stay unaffordable; it became a decisive and enormously capitalised input, because RLHF and evaluation need it and no amount of raw web text substitutes. A reading meeting a data-labour story — annotator pay, contractor conditions, an evaluation-set scandal — has a dated baseline for how thoroughly that expectation reversed.

Fourth: the Semantic Web section is a reusable diagnostic for standards announcements. The four obstacles — long-tail cost, implementation difficulty, competition between factions, and deception — are a checklist that applies with almost no translation to every proposed interoperability layer that has since been announced for agents: tool schemas, agent-to-agent protocols, content-provenance standards, watermarking registries. "Every ontology is a treaty" is the sentence to reach for when a consortium announces a shared format and the members are competitors. A reading meeting such an announcement can ask the 2009 question — is there an established, honest, self-correcting group of cooperative users here, or is there competition and deception? — and get a useful answer without predicting anything.

What this entry does not support. It is not evidence that scaling works, that scaling has stopped working, or that any particular model is over- or under-trained; it predates every system now under discussion and contains no measurement of one. It does not support "data is running out" as a claim in its own right — that is Epoch's estimate, with its own uncertainty, and this entry's contribution is only the dated before picture. And it is not a needle argument in either direction: the same arc reads as AItopia (the recipe worked so well it consumed its own input) or AImageddon (the commons is being enclosed and strip-mined) depending on evidence this file does not contain.

What it got right, and what it got wrong

All claims below were made in the March/April 2009 issue, on the record, with no due date attached by the authors. Where a claim came due at a datable event, I give the event.

Claim 1 — "invariably, simple models and a lot of data trump more elaborate models based on less data." Split, and the split is the most interesting thing in the paper.

The data half: right, to a degree the authors did not imagine. They were arguing from a trillion words. Frontier training sets crossed ten trillion tokens by 2024, and the essay's central intuition — that crossing a data threshold changes what an unchanged method can do — is now the field's most reliable empirical regularity.

The "simple models" half: wrong, and the word invariably is what makes it gradeable. The authors define their own terms, so the grade does not depend on mine: in this paper "simple" means n-gram counting with Good-Turing or Kneser-Ney smoothing, memorized phrase tables, and "simple n-gram models or linear classifiers based on millions of specific features." Against that definition, elaborate models based on the same or more data won, and won completely. A 2026 frontier system is a deep transformer with attention, learned positional structure, sparse routing, and a multi-stage post-training pipeline. It is not counting.

The fair reading of the miss is that they conflated two axes that came apart: elaborate in hand-supplied knowledge (grammars, ontologies, linguistic rules) and elaborate in learned structure (depth, attention, learned representations). The first kept losing, exactly as predicted. The second won, and the paper had no category for it. Sutton's 2019 restatement is more durable precisely because it draws the line at human knowledge rather than at simplicity: a transformer is a general method that scales, and it is not simple.

Claim 2 — "For many tasks, words and word combinations provide all the representational machinery we need to learn from text." Wrong, and superseded from below as well as above.

Two things ate this. From above, learned continuous representations replaced word combinations as the unit of computation. From below, the word itself stopped being the unit: sub-word tokenisation (Sennrich, Haddow and Birch, ACL 2016, on rare words and byte-pair encoding) means production systems do not have a "word" primitive at all. The essay's advice to "see how far you can go by tying together the words that are already there" named the exact ceiling the field went through in both directions.

Claim 3 — "Abstract representations (such as clusters from latent analysis) that lack linguistic counterparts are hard to learn or validate and tend to lose information." Wrong in its conclusion, right in its premise, and the sharpest miss in the paper.

Abstract representations lacking linguistic counterparts are now the entire stack: word2vec (Mikolov et al., January 2013), contextual embeddings, and the transformer's residual stream, in which no dimension has a linguistic counterpart and the model's competence lives anyway. Four years and one building. word2vec came out of Google, four years after three Google researchers advised the field against exactly that class of representation.

Two qualifications, both of which favour the authors.

The premise stands. "Hard to learn or validate" was correct and remains correct; the field spawned an entire subfield — mechanistic interpretability — around the difficulty, and that subfield's existence is a concession that the 2009 objection identified something real. What was wrong was the inference: therefore prefer representations with linguistic counterparts. The right inference was therefore expect validation to be a permanent open problem, and proceed anyway.

And one of the three authors partly retracted it in public, two years later. In "On Chomsky and the Two Cultures of Statistical Learning" (2011), defending statistical methods against Chomsky's charge that they yield no insight, Norvig writes: "I agree that it can be difficult to make sense of a model containing billions of parameters... But one can gain insight by examining the properties of the model." And: "engineering success shows that something is working right, and so is evidence (but not proof) of a scientifically successful model." That is a defence of exactly the opaque, non-linguistic representation the 2009 paper had warned against — written before the models that would vindicate the reversal existed. It is the closest thing to an author's self-grade this entry found, it is a partial reversal rather than a confirmation, and no author has, as far as this session could find, published a formal retrospective grading of the 2009 claims.

Claim 4 — "Choose a representation that can use unsupervised learning on unlabeled data, which is so much more plentiful than labeled data." Right, and it is the paper's best call by a wide margin.

This is the recipe. Self-supervised pretraining on unlabeled text is what GPT-1 (Radford et al., June 2018) and BERT (Devlin et al., October 2018) did, and what every frontier model since has done. Nine years early, stated as a design instruction, and correct. If this entry survives one sentence, it is this one.

The irony is that this correct prescription is what consumed the resource the same paragraph called plentiful — see claim 7.

Claim 5 — "Represent all the data with a nonparametric model rather than trying to summarize it with a parametric model, because with very large data sources, the data holds a lot of detail." Wrong on the letter, partially rescued as a component.

The winning systems are emphatically parametric and do exactly the summarising the sentence warns against: a model compresses trillions of tokens into on the order of 10¹¹–10¹² weights. That is the summarisation the authors said not to do, at a ratio of roughly a hundred to one or worse.

Two honest rescues. First, the instinct survives as an architectural component rather than as the architecture: retrieval-augmented generation, vector stores and long-context document stuffing are nonparametric memory bolted onto a parametric model, and the motivation given for them — that the parametric model loses detail and the tail — is the 2009 sentence's motivation verbatim. Second, at extreme scale, large parametric models behave partly nonparametrically: they memorise the tail rather than smoothing it, which is why claim 6 held. But the field did not follow this instruction, and the reason is one the paper could not have had: inference cost. A nonparametric model that holds all the data is expensive at query time, and every economic pressure since 2020 has run toward the opposite.

Claim 6 — "memorization is a good policy if you have a lot of training data." Right, and it became a legal liability the authors had no reason to anticipate.

Memorisation held technically — large models do memorise, and the memorisation of the long tail is part of what makes them useful. But the sentence is now the technical fact underneath the field's largest legal exposure, because a model that memorises its training data can reproduce it. In Bartz v. Anthropic, Judge William Alsup held on 23 June 2025 that training on lawfully acquired books was fair use and "quintessentially transformative," while separately holding that downloading pirated copies was not — a distinction that turns entirely on acquisition, not on what the model does with the text. Anthropic settled the piracy class action for $1.5bn in late August 2025, preliminarily approved that September and finally approved on 20 July 2026, the largest copyright recovery in US history. This is a "right for reasons that became a bill" grade: the engineering advice was sound and the ledger it landed in was not the one the authors were writing about.

Claim 7 — the unstated premise: data is abundant, cheap, growing, and yours for the gathering. Inverted. This is the entry's payload.

The paper never argues this claim because in 2009 it needed no argument. The web was growing, crawling was free, nobody had sued anyone, and the scarce input was the annotation you avoided needing. Seventeen years later every clause has flipped:

The honest qualification. "Inverted" describes the premise, not the conclusion. Nothing above shows the data-scaling thesis failed — it shows it succeeded so thoroughly that it consumed its input and provoked its own enclosure. And the exhaustion estimate is a projection with a wide interval, not an observed fact; a reading citing it should cite the interval.

Claim 8 — Semantic Web: it will "work best where an established, honest, self-correcting group of cooperative users exists and not as well where competition and deception exist," and semantic interpretation must be learned from data regardless. Right, and it is the best-aged prediction in the paper after claim 4.

The formal-ontology web did not arrive. What did arrive, in June 2011 — two years later, from Google, Bing and Yahoo!, with Yandex joining that November — was schema.org: a single deliberately shallow vocabulary imposed by the parties who could make publishers care, succeeding exactly where the incentive to cooperate was real (publishers wanting rich search results) and drawing spam wherever it was not. That is the 2009 prediction's shape, including the failure mode.

The strongest confirmation is the second half. The semantic interpretation problem was in the end addressed — to whatever degree it has been addressed — by learning from unlabeled text, precisely as the paper said it would have to be, and not by anybody agreeing on ontologies.

The partial counterexample, recorded because omitting it would flatter the entry. Google's Knowledge Graph (2012) is a large curated structured knowledge base and it did become central to search. But it was assembled by extraction and acquisition — Freebase, Wikipedia, web-scale table and text mining, the WebTables lineage this very paper points at — rather than by the world publishing ontologies. The authors were wrong that structured knowledge would not matter and right about where it would come from.

Scorecard

Eight graded items: three right (4, 6, 8), three wrong (2, 3, 5), one split (1), and one inverted premise (7) that no one graded at the time because no one stated it. A ninth, ungraded above because it is a lesson rather than a claim — "much Web data consists of individually rare but collectively frequent events" — was right, and is the one the synthetic-data failure mode in claim 7 now threatens from a direction the authors could not have seen. The prescription that mattered most — unsupervised learning on plentiful unlabeled data — was correct nine years early. The prescriptions about how to represent were wrong, uniformly and in the same direction: they underrated learned structure with no linguistic counterpart. A field that had followed this paper's representational advice literally would have spent the 2010s building better n-gram models.

Commonly misused as

Not required for interpretation, and included because this paper is misused in four distinct ways, three of them by people who have only met the title.

As "more data always beats a better model." The paper says simple models and a lot of data beat more elaborate models based on less data — a comparison at unequal data, not a claim that model quality is irrelevant at equal data. The literal reading was falsified by 2017 at the latest and is contradicted by the entire architecture literature. scaling-laws-2020 is the entry that makes the actual tradeoff quantitative, and it shows both terms mattering.

As an early statement of the Bitter Lesson. It is an ancestor, not a version. The Bitter Lesson's ally is compute; this paper's ally is data, and compute is almost absent from its five pages. Between 2020 and 2022 that difference was worth billions of dollars in misallocated training runs: Kaplan's compute-optimal rule pointed at parameters, and Chinchilla (Hoffmann et al., 2022) corrected it toward data — roughly twenty tokens per parameter against GPT-3's under two — which is a vindication of this paper against a compute-first reading. Collapsing the two doctrines erases the axis on which one of them was right.

As a licence for data maximalism — "quality doesn't matter, just get more." The paper's actual position is narrower and better: unfiltered data is worth its errors when the size ratio is a million to one, which is what they say about the trillion-word corpus against the Brown Corpus. It is a statement about a specific ratio at a specific threshold, not a general dismissal of curation. Modern practice — heavy deduplication, quality filtering, curriculum ordering — is not a refutation of this paper so much as a demonstration that the million-to-one ratio was a one-time windfall.

As Norvig's paper. The id says norvig-data-2009 and the id is wrong about the authorship even though it is the id in use. Alon Halevy is first author, Fernando Pereira third, Norvig in the middle; the conventional short cite is "Halevy, Norvig and Pereira." The id survives because it is what the canon chain was given and because Norvig is the name the argument travelled under, which is itself a small instance of how attribution drifts toward the famous name. Nothing in the file depends on it, and the header block names all three.

Sources

Read directly, in full:

Reached through search summaries, not read in the original. Each is used only for the one fact attributed to it, and a reading that wants to lean on any of them should open the primary:

Not consulted, and named because their absence is a real limit: the IEEE Intelligent Systems issue's surrounding "Expert Opinion" material and any contemporary 1970s-style rebuttal; Alon Halevy's and Fernando Pereira's own later writing, which may contain retrospectives this file does not know about.

In this canon, and read directly for this entry: bitter-lesson-2019 (which named this paper and prompted the id), scaling-laws-2020, samuel-checkers-1959, expert-systems-collapse-1987, deep-blue-1997, sparse-moe-2017, alexnet-2012, transformer-2017, goodharts-law-1975, lighthill-1973.

This entry places no needle, no score and no landmark. Nothing in it is evidence, and nothing in it may be deposited into history/events.jsonl.