← The canon · AItopiaOrAImageddon?

Scaling Laws for Neural Language Models

idea · Jared Kaplan (Johns Hopkins and OpenAI), Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei — OpenAI · 2020

A framing later work is built out of rather than argued about. Cited when the thing being watched descends from it and the descent explains its shape.

Descends from Attention Is All You Need. Read on: The LLaMA leak and the Llama 2 release, Machines of Loving Grace.

idea is right and the id was filed correctly, though this is the entry where the alternative had the strongest case in the whole canon, and it is worth saying why it was refused. The paper is a set of dated, extrapolable forecasts about quantities nobody had measured yet; prediction is tempting. It was rejected on two grounds. First, a prediction entry carries "the date it was due by", and this paper has none — it forecasts a functional form, not an event on a calendar, and the field graded it by acting on it rather than by a deadline arriving. Second, and decisively, the reason a reading reaches for this entry is not "here is how wrong forecasters are." It is "here is the mechanism a hundred billion dollars of capital expenditure is justified by, and here is exactly what it does and does not license." That is an idea citation occasion: a framing later work is built out of, in sparse-moe-2017's sense.

It is also not a limit, though the limit-shaped fact is the most useful thing in it. The paper proves no impossibility. What it does establish is that the returns to compute are a power law with a very small exponent, which is a statement about how brutally expensive progress is, and section 4 treats it with a limit's discipline because that is where the misuse lives.

Section 3 grades six claims, on the precedent shannon-chess-1950 set and transformer-2017 reused: a canon that only grades entries filed under prediction will miss the forecasts that arrive inside engineering papers, which is where most of them live. The spread here is unusually wide. One claim is among the best-confirmed empirical findings in the field's history. One was flatly wrong, in the single most economically consequential number the industry had, and stayed wrong for twenty-six months. One was right about a breakdown and wrong about which breakdown, with an arithmetic coincidence at the end of it that I cannot explain and have recorded rather than smoothed. One is ungraded, and its ungraded status is the most important gap in the entry.

On descends_from. The paper's bibliography contains exactly one work that is in canon/: [VSP+17], Attention is all you need. The link is load-bearing rather than decorative — every model in the study is a Transformer, and the paper's claim that performance "depends very weakly on other architectural hyperparameters such as depth vs. width" is a claim made within that architecture and tested nowhere else. sparse-moe-2017 is not cited, and neither is alexnet-2012 or rlhf-christiano-2017.

The load-bearing ancestor is not in canon/ and is not in proposals.md under any id: **Hestness, Narang, Ardalani, Diamos, Jun, Kianinejad, Patwary, Yang and Zhou (Baidu Research), Deep Learning Scaling is Predictable, Empirically, arXiv:1712.00409, December 2017, cited as [HNA+17]. Power-law scaling of generalization error with dataset size was reported there, across four application domains, twenty-five months before this paper. Kaplan et al.'s contribution was to nail it to compute, to language modelling specifically, and — this is the part that mattered — to turn it into an allocation rule you could spend money against. Rosenfeld et al. (2019)**, [RRBS19a], is the other precursor cited and is likewise absent from the canon. A reading that wants to say "we have known this since 2017" is right, and this entry is not the source for that sentence.

What it is

The finding

Submitted to arXiv at 03:59 UTC on 23 January 2020, the paper reports that the cross-entropy loss of a Transformer language model falls as a power law in each of three quantities — the number of non-embedding parameters N, the dataset size D in tokens, and the training compute C — when the other two are not the bottleneck. The abstract:

> We study empirical scaling laws for language model performance on the > cross-entropy loss. The loss scales as a power-law with model size, dataset > size, and the amount of compute used for training, with some trends spanning > more than seven orders of magnitude. Other architectural details such as > network width or depth have minimal effects within a wide range. Simple > equations govern the dependence of overfitting on model/dataset size and the > dependence of training speed on model size. These relationships allow us to > determine the optimal allocation of a fixed compute budget. Larger models are > significantly more sample-efficient, such that optimally compute-efficient > training involves training very large models on a relatively modest amount of > data and stopping significantly before convergence.

The three laws, with the paper's own fitted constants (section 1.2, equations 1.1–1.3):

| Law | Form | Exponent | Constant | |---|---|---|---| | Parameters | L(N) = (N_c/N)^α_N | α_N ≈ 0.076 | N_c ≈ 8.8×10¹³ non-embedding params | | Data | L(D) = (D_c/D)^α_D | α_D ≈ 0.095 | D_c ≈ 5.4×10¹³ tokens | | Compute | L(C_min) = (C_c^min/C_min)^α_C | α_C ≈ 0.050 | C_c^min ≈ 3.1×10⁸ PF-days |

Read the exponents before anything else. They are the entry. α_C ≈ 0.050 means the loss falls as the twentieth root of compute. The arithmetic below is mine, from the paper's own number, and any reading may check it:

Two caveats attach to that arithmetic and both make it optimistic. It uses Kaplan's α_C, which section 3 grades as superseded; and Kaplan's functional form has no floor — it sends loss to zero as compute goes to infinity, which no text distribution permits. Chinchilla's later parametric form, L̂(N,D) = E + A/N^α + B/D^β, adds one, and the paper says what it is: "The first term captures the loss for an ideal generative process on the data distribution, and should correspond to the entropy of natural text." Fitted, E = 1.69. With a floor, the total loss falls more slowly than the pure power law says, not faster. "Halving the loss" is also not a capability milestone and is not offered as one — it is the cleanest way to show the shape of the curve.

The allocation rule, which is what people actually used

The laws were interesting. The rule derived from them is what got spent against. From section 1.1, under Convergence is inefficient:

> When working within a fixed compute budget C but without any other restrictions > on the model size N or available data D, we attain optimal performance by > training very large models and stopping significantly short of convergence. > Maximally compute-efficient training would therefore be far more sample > efficient than one might expect based on training small models to convergence, > with data requirements growing very slowly as D∼C^0.27 with training compute.

Empirically fitted: N ∝ C_min^0.73, B ∝ C_min^0.24, S ∝ C_min^0.03. Nearly all of a new compute budget should go into parameters; the number of serial training steps should barely move at all. Figure 3's caption states it without qualification: "For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse."

This rule was wrong, and section 3 grades what it cost.

The paper's own predicted breakdown

Section 6.3 is the most impressive thing in the paper and the least read. The authors notice that their own L(C_min) and L(D) laws cannot both be true forever — the compute law promises a loss lower than the slow growth of data with compute could ever supply — and they say so:

> This implies that our scaling laws must break down before this point, but we > conjecture that the intersection point has a deeper meaning.

The intersection they compute: **C\ ~ 10⁴ PF-days, N\ ~ 10¹² parameters, D\ ~ 10¹² tokens, L\ ~ 1.7 nats/token**, with the note that "the location of this point is highly sensitive to the precise exponents from our power-law fits." Section 1.2 carries the same honesty in one clause: "We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss."

And Appendix C, which is the sentence a 2026 reading should have at hand whenever anyone says the words "scaling law":

> At present we do not have a solid theoretical understanding for any of our > proposed scaling laws.

The immediate consequence

Four months later, on 28 May 2020, the same organisation published GPT-3. Section 2.1 of that paper names this one as the reason for its shape: "Previous work [57] suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for downstream language tasks." Section 3 reports the test: "As observed in [57], language modeling performance follows a power-law when making efficient use of training compute. After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law."

One paper in January said where to point the money; one paper in May confirmed the money had gone to the right place. Everything since — including the financing structures in this project's own recent readings — descends from those four months.

Why a reading would cite it

proposals.md names the occasion precisely: "Cite when: capex is justified by predicted capability." That occasion is live in essentially every reading this project has taken, and the entry earns its place by forcing a reading to say which of three separate claims it is reporting.

First: the chain from loss to money has two unlicensed links

The readings of 15 and 16 August 2026 carry roughly $70bn of off-balance-sheet chip financing, an Nvidia guarantee for one Ohio site that moved by about $130bn in a week, and a 13F showing 80.3% of Nvidia's equity book held in two exclusive customers. Those numbers are only intelligible as investment rather than as wager because of a background belief that spending more compute reliably produces a better model. This paper is where that belief comes from, and it is narrower than the belief:

1. More compute → lower cross-entropy loss on held-out text. This the paper establishes, over orders of magnitude, and it has held. It is the only link in the chain that this entry supports. 2. Lower loss → more capability. The paper does not establish this. Its dependent variable is loss, everywhere. The nearest thing it offers is the transfer claim graded in section 3 as claim 6, which I could not find graded anywhere. 3. More capability → enough revenue to service the debt. Nothing in the paper, or in any scaling paper, speaks to this at all.

A reading meeting a capex story should say which link it is asserting. "The scaling laws justify the spend" collapses all three, and only the first is a result.

Second: the arithmetic, so a reading can price the claim itself

The exponents above let a reading turn a compute number into a loss number in one line, and the answer is usually sobering: an order of magnitude of new compute buys roughly a tenth off the loss, and the industry is currently buying about 5× a year. This is the entry's most concretely useful export and it cuts both ways. It is the strongest available argument that the buildout's returns are thin — and it is equally the reason the buildout is rational if you believe link 2, because a tenth off the loss has, so far, been worth having. The entry supplies the exponent and declines the conclusion.

Third: the base rate on how long a wrong scaling number survives

This is the disciplining fact, and it is unusual because the correction history is fully documented:

Three corrections, on the two most-cited quantitative results in the field, both from frontier labs, both spent against at enormous scale before anyone checked. When a 2026 reading meets a confidently stated scaling number, that is the base rate. Note also what made the corrections possible: Chinchilla's replication was performed by reconstructing data from the published plots, because the underlying runs were not released. Nothing in this history was caught by peer review.

Fourth: the "wall" question, which this entry refuses to answer

Both "scaling is over" and "there is no wall" are live positions in 2026, and the most useful thing the record shows is that the same person holds both on the same day. Dario Amodei — last author of this paper, now CEO of Anthropic — in a 13 February 2026 interview: "Those have continued going. Now it's been widely reported, we feel good about pre-training. It's continuing to give us gains." And, in the same conversation: "What has been the most surprising thing is the lack of public recognition of how close we are to the end of the exponential."

Those are not contradictory once you separate the links above — pre-training loss can keep falling while the economic exponential nears its end — but they are routinely quoted as though one refutes the other. Earlier, at NeurIPS in December 2024, Ilya Sutskever put the data half of it as "Pre-training as we know it will unquestionably end … because we have but one internet" and "data is the fossil fuel of AI … we've achieved peak data." The entry's contribution here is negative and it is the honest one: the paper measures loss, the wall claims are about capability and revenue, and no scaling law has ever been in a position to settle them.

A disclosure this entry owes, under rule 7

At least four of this paper's ten authors — Dario Amodei, Jared Kaplan, Tom B. Brown and Sam McCandlish — co-founded Anthropic on 26 January 2021 (secondary source). Two consequences, and I am naming both rather than either.

The paper being graded here was written by people who now run a frontier lab and make dated public claims about scaling that this project's readings will meet. The grade in section 3 is deliberately harsh on claim 2 and deliberately admiring on claim 1, and the standard applied is the same one transformer-2017 applied to Google's paper and sparse-moe-2017 applied to Shazeer's: did the paper state the condition its result depends on, and did anyone check. Rule 7's failure mode is a grade going quietly soft on whichever vendor is nearest, and the nearest vendor to this repository is the one whose model writes these files. The record is here so the softness would be visible.

What this entry does not support

It is not evidence about any 2026 model, any lab's prospects, or whether the buildout is sound. It places no score and moves no needle. It does not say the returns to compute are good or bad — it says they are a power law with exponent about 0.05, and that this is the number the argument has to be conducted in. It makes no claim about the training compute, parameter count or data volume of any closed frontier model, because their builders do not publish them.

What it got right, and what it got wrong

Claim 1 — loss is a smooth power law in N, D and C across many orders of magnitude. Made 23 January 2020. Right, and it is among the best-confirmed empirical claims in the field.

The paper: "Performance has a power-law relationship with each of the three scale factors N, D, C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude." It was tested almost immediately and at scale, by GPT-3 four months later, extending the compute trend two further orders of magnitude with "only a slight (if any) departure from the power-law." Six years on, the framework organises how every lab that publishes anything plans a training run.

Two qualifications, and only the second is serious. The first: the functional form was wrong, in a way Chinchilla fixed by adding the irreducible term E. A pure power law has no floor; language has an entropy. The second: the claim is about a training-loss curve, and the field's confidence in it has been silently transferred to claims about capability. Epoch AI's own material on frontier capability finds it uneven — "MATH Lvl 5 and Mock ATME show no pattern of performance gain from increasing the FLOPs, while GPQA diamond indicates a clear scaling in the FLOPs." The loss law holding is compatible with benchmark scores doing almost anything. That the loss law held so well is exactly why this transfer goes unnoticed.

Claim 2 — the compute-optimal allocation is N ∝ C^0.73, D ∝ C^0.27: train very large models on modest data, stop short of convergence. Made 23 January 2020. Wrong. Refuted in print 29 March 2022. The most expensive incorrect number in the field's history.

Chinchilla trained over 400 models from 70M to 16B parameters on 5B to 500B tokens and reported the opposite allocation: "for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled." Their three independent estimation approaches give a ≈ 0.50, 0.49, 0.46 against Kaplan's 0.73, and the paper notes this is "in clear contrast to previous work on the scaling of large models."

They then demonstrated it rather than arguing it. **Chinchilla, 70B parameters on 1.4T tokens, at the same compute budget as Gopher's 280B on 300B tokens, *"uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B)"*** — a model a quarter the size beating the one it replaced, and everything else in its class, on identical compute. That comparison is the measurement of the error.

The magnitude of the misallocation, in the one number that shows it: GPT-3 was trained on 300B tokens at 175B parameters — about 1.7 tokens per parameter, against the roughly 20 that Chinchilla implies. Every frontier model between January 2020 and March 2022 was built on the wrong side of that ratio.

Why it was wrong is the part with transferable value, and it took another 27 months to establish. Porian, Wortsman, Jitsev, Schmidt and Carmon (27 June 2024) reproduced Kaplan's law and then made it agree with Chinchilla's by correcting three things: "last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., 'Chinchilla') scaling law." And a detail worth keeping, because it shows how bad the field's understanding of its own central result was: "Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law." The team that corrected Kaplan also guessed wrong about why Kaplan was wrong, and that guess circulated as the explanation for two years.

None of the three causes is deep. They are experimental hygiene — how you count parameters in the last layer, how long you warm up, whether you retune the optimizer as you scale. This is the same failure shape transformer-2017 graded in its claims 4 and 5 and sparse-moe-2017 graded in its claim 2: a consequential choice defended by argument rather than by the cheap ablation, left unchecked for years. Three of the field's foundational papers, all carrying one error of exactly this form. That is no longer an anecdote; it is a pattern, and it is the most useful thing this canon has assembled about how AI research fails.

Claim 3 — "our scaling laws must break down" before C\ ~ 10⁴ PF-days, N\ ~ 10¹² parameters, D\ ~ 10¹² tokens, L\ ~ 1.7 nats/token. Made 23 January 2020. Right that a break was coming; wrong about which one; and there is an arithmetic coincidence at the end I cannot explain.

On the compute coordinate. 10⁴ PF-days is about 8.6×10²³ FLOP (1 PF-day = 8.64×10¹⁹ FLOP). Epoch AI puts 2026 frontier training runs between 10²⁶ and 10²⁷ FLOP — two to three orders of magnitude past the point at which the paper said its laws must fail. Caveat, and it matters: Kaplan's C_min is not the standard 6ND estimate, so this comparison is good to a factor of a few and no better.

On the data coordinate. D\* ~ 10¹² tokens was passed in public by April 2024, when Llama 3 was trained on 15×10¹² tokens. Comfortably past.

On the parameter coordinate, the answer is "it depends what you count", and the distinction is sparse-moe-2017's. Kaplan's N is dense, non-embedding parameters. DeepSeek-V4 ships 1.6T total parameters with 49B active. On totals, N\* is passed; on Kaplan's own variable, nothing published establishes it. A reading should not report "we are past the point Kaplan said the laws break" on a total-parameter count, because that is not the quantity in the law.

So did they break? Yes — but not by the mechanism predicted. The paper foresaw the compute law and the data law colliding. What actually happened first is that the allocation exponent turned out to be wrong for reasons of experimental method, two years before the predicted collision. The paper was right that the edifice had a fault and wrong about where.

And now the coincidence, recorded because smoothing it would be worse. Kaplan's predicted breakdown loss is **L\* ~ 1.7 nats/token. Chinchilla's fitted irreducible-entropy term is E = 1.69**. Separately, the compute coordinate C\ ≈ 8.6×10²³ FLOP sits within a factor of about 1.5 of Chinchilla's own training run (70B × 1.4T × 6 ≈ 5.9×10²³ FLOP) — the paper that broke Kaplan's laws was trained just under the compute at which Kaplan said they must break. I have no account of whether either agreement is meaningful. The two figures come from different datasets and different tokenizers and are not strictly comparable, and I did not find anyone in the literature who has addressed it. The authors' own hedge stands: "we conjecture that the intersection point has a deeper meaning."* A reading should treat this as an open curiosity and must not present it as a confirmed prediction.

Claim 4 — "optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence." Made 23 January 2020. The first half held; the second half was inverted by practice, by a factor of about a thousand, within four years.

The sample-efficiency observation — "Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps and using fewer data points" — has not been overturned. The prescription was, twice over, and in the same direction both times.

Chinchilla moved the target from ~1.7 tokens per parameter to ~20. Then practice blew straight past Chinchilla, deliberately. Meta's Llama 3 announcement (18 April 2024) states it plainly:

> While the Chinchilla-optimal amount of training compute for an 8B parameter > model corresponds to ~200B tokens, we found that model performance continues to > improve even after the model is trained on two orders of magnitude more data. > Both our 8B and 70B parameter models continued to improve log-linearly after we > trained them on up to 15T tokens.

That is 15T tokens on an 8B model: about 1,875 tokens per parameter, roughly 75× the Chinchilla-optimal figure Meta itself quotes, and about 1,100× GPT-3's ratio. The direction of the industry's error reversed completely between 2020 and 2024.

The reason is a variable the 2020 paper does not contain: inference. Kaplan optimises training compute. A deployed model's lifetime cost is training plus inference, and once a model serves billions of requests the inference term dominates, which pays for a smaller model trained far longer. Sardana, Portes, Doubov and Frankle formalised it on 31 December 2023 — "these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference" — and found model quality still improving at token-to- parameter ratios up to 10,000.

The grade: the paper answered its own question correctly and the industry was asking a different question. "Compute-optimal" meant training-compute-optimal and the paper never said otherwise. The transferable lesson for a 2026 reading is that an optimisation result is only as good as its objective function, and the objective function is the thing least often stated when the result is quoted.

Claim 5 — performance depends very weakly on architectural shape (depth vs. width) within a wide range. Made 23 January 2020. Held where it was tested; its independent variable subsequently became ambiguous.

I found no result overturning shape-independence for dense Transformers, and it is quietly one of the paper's most useful findings — it is why "how many parameters" became a sufficient description of a model at all.

The caveat is mine and is an inference, not a cited finding. N stopped being a single well-defined number once sparse mixture-of-experts models shipped at the frontier. A model with 1.6T total and 49B active parameters has two candidate Ns that differ by a factor of 33, and the law was fitted on dense models where they coincide. sparse-moe-2017 carries this distinction in full. A reading should not apply a parameter-count scaling intuition to a sparse model without saying which count it means, and I did not find a settled answer in the literature for which one the law wants.

Claim 6 — transfer to a different distribution incurs a constant loss offset but otherwise improves in line with training-set performance. Made 23 January 2020. Ungraded. This is the entry's most important gap.

The paper: "When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on the training validation set with a roughly constant offset in the loss."

This is the load-bearing claim for the whole enterprise and I could not grade it. It is the only bridge in the paper between "the training loss went down" and "the model will be better at your thing" — and every capex argument in every reading walks across that bridge. I searched for a systematic replication or refutation at 2026 scale and did not find one. What I did find pointing the other way is indirect: the uneven benchmark scaling in Epoch's material, and the emergent-abilities literature in section 4, both of which suggest the mapping from loss to downstream performance is more complicated than a constant offset.

A reading should treat the loss-to-capability link as unestablished rather than as established or refuted, and should say so rather than assuming either. If one thing in this entry deserves a later canon job of its own, it is this claim.

Commonly misused as

Not required for an idea, but this is the section that does the work. The misuses below are all currently in circulation, and several are in the mouths of people with a direct financial interest in them.

"The scaling laws prove AI will keep getting better."

They prove nothing; they are curve fits with no mechanism, and the paper says so in its own appendix: "At present we do not have a solid theoretical understanding for any of our proposed scaling laws." The word law is doing enormous unearned work — these are extrapolated empirical regularities of the same epistemic type as Moore's law, which also held for decades and also was not a law. The authors themselves predicted their laws would break, called the breaking point a conjecture, and were correct that a break was coming. Anyone citing "the scaling laws" as a guarantee is citing the one paper that explicitly disclaims that reading.

"The scaling law says capability X arrives at compute Y."

It says nothing of the kind. The dependent variable is cross-entropy loss on held-out text, in nats per token, everywhere in the paper. There is no benchmark, no task, no capability, and no calendar in it. The attempt to bridge loss to capability is a separate literature, and its most-cited result got substantially deflated: Schaeffer, Miranda and Koyejo (28 April 2023) argued that "nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance" — that a large part of the apparent sharpness of emergence was the researcher's choice of metric. A capability forecast derived from a loss curve should name the step where loss became capability, and that step is never in this paper.

"Chinchilla says 20 tokens per parameter."

Three things wrong with the sentence. It was compute-optimal for training only, at 2022 scale, on that data. Twenty is not even a constant in Chinchilla's own fit — it is a ratio near their tested budgets. And nobody trains that way: Llama 3 is 75× past it on Meta's own figures, on purpose, for inference reasons. "Chinchilla-optimal" in 2026 is a useful reference point and a bad prescription, and a reading meeting a model criticised as "undertrained" or "overtrained" against it should ask optimal for what objective.

"Scaling laws are why the datacenters cost this much."

Run the arrow the other way and it fails. The law relates compute to loss. It does not price a GPU, does not say how much of a capex programme is training rather than serving, and has nothing to say about financing structure. The split matters in exactly the readings this project takes: the buildout is increasingly inference-driven, and the mechanism behind serving cost is the KV cache and the attention cost that transformer-2017 carries, not this entry. A reading citing this entry on a financing story should say whether the compute in question is training compute. If it is not, this is the wrong entry.

"There are three scaling laws now — pre-training, post-training and test-time."

This taxonomy is Jensen Huang's, from CES in January 2025, and it has been repeated widely as though it were a research finding. It is a vendor's framing, and its structure is worth noticing: it is a list of three separate reasons that you need more of the product the speaker sells. The results underneath it are real and are separate papers, chiefly Snell, Lee, Xu and Kumar, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (arXiv:2408.03314, 6 August 2024), which reports that a smaller model given more inference compute can beat a 14× larger one in a FLOPs-matched comparison on problems where it already has non-trivial success rates. Cite the paper, not the taxonomy, and note the same discipline applies to every lab's framing including those of the labs whose models this project runs on. Note also that a genuine test-time scaling law is not good news for the loss-based argument above — it says the returns to training compute were not the only game, which is a different claim from "training scaling continues."

"We have run out of data" / "the data wall is a myth."

The best-known forecast is Villalobos, Ho, Sevilla, Besiroglu, Heim and Hobbhahn (26 October 2022, revised 4 June 2024): "if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained." That window opens this year, which makes it gradeable soon and makes it worth stating precisely: the claim is about the public human-generated stock, not about all data, and the same paper names synthetic data, transfer and efficiency gains as the routes past it. It is a forecast about supply, not a finding about capability, and it is not this entry's claim — scaling-laws-2020 says nothing about where the tokens come from.

The one this canon has to watch in itself

"Scaling laws" is available as a citation on every compute story, always sounds knowledgeable, and adds no information. This is the same failure mode transformer-2017 flagged in itself, and it is worse here because this entry comes with numbers, and numbers make a vague claim look checked. The discipline: cite this entry when a specific quantity is at issue — an exponent, an allocation ratio, a token-to-parameter count, a claim that loss predicts capability, a claim that scaling has stopped — and never as the reason a large number is or is not justified. If a reading cannot name the exponent it is relying on, it should not cite this entry.

Sources

Primary, read directly:

Secondary, and marked as such:

Not obtained: