Attention Is All You Need

idea is right and the id was filed correctly, in the same sense sparse-moe-2017 established: a framing that later work is built out of rather than argued about, where the citation occasion is "the thing you are watching descends from this, and the descent explains its shape." It is not a moment — there was no adjudicated public event, no withheld test set, no rival field arriving to be beaten. It was a NeurIPS paper, and the field's recognition of what it had was gradual rather than dated. It is not a limit: it proves no impossibility, though section 3 grades a complexity claim that has since behaved rather like one.

It is not a prediction, and section 3 grades its forecasts anyway, on the precedent shannon-chess-1950 set and sparse-moe-2017 reused: a canon that only grades entries filed under prediction will systematically miss the forecasts that arrive embedded in engineering papers, which is where most of them live. This paper carries six dated, checkable claims. Two were refuted in print, one is still open after nine years, and one is the most comprehensively vindicated architectural claim in the field's history. The spread is the entry's value.

A word on descends_from, because the link is real but thin and overstating it would be worse than omitting it. The paper's bibliography contains exactly one work that is in canon/: reference [32], "Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017" — that is sparse-moe-2017, published five months earlier with Shazeer as first author, who is second author here. The in-text citation is in the introduction:

> Recent work has achieved significant improvements in computational efficiency > through factorization tricks [21] and conditional computation [32], while also > improving model performance in case of the latter.

That is a background nod, not a foundation. Nothing in the Transformer is built out of the mixture-of-experts layer; the sentence is acknowledging neighbours in a paragraph about reducing sequential computation. The link is listed because it is documented, dated and in the introduction, and because the two entries are genuinely a pair — but a reading should not infer that the Transformer came out of MoE. It did not.

The relationship between the two entries runs the other way, and is worth holding straight. sparse-moe-2017 names this paper as one of its own three proximate ancestors and grades what its arrival did: the MoE results were obtained on stacked LSTMs, and 140 days after the MoE paper hit arXiv its first author was second author on this one, which retired that substrate. So the citation points backward from Transformer to MoE, and the consequence points forward from Transformer to MoE. They are siblings five months apart sharing an author, and the modern frontier is the two of them composed — a Transformer whose position-wise feed-forward sublayer has been replaced by a sparse mixture. Read together they explain nearly the whole shape of a 2026 open-weight release.

The load-bearing ancestors are three, and none of them is in canon/:

• Bahdanau, Cho and Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate, reference [2]. This is where attention comes from. The 2017 paper's title is a claim about this mechanism — that it suffices alone — and is unintelligible without it. Not proposed under any id.
• Hochreiter and Schmidhuber (1997), Long Short-Term Memory, reference [13] — the thing being dispensed with. Not proposed.
• Gehring, Auli, Grangier, Yarats and Dauphin (2017), Convolutional Sequence to Sequence Learning, reference [9] — the other contemporary attempt at the same goal, and the direct comparison the paper's Table 1 is constructed to win. Not proposed.

scaling-laws-2020 and rlhf-christiano-2017, both proposed and unwritten, are the two entries that with this one would explain how a 2026 model was built at all: this one supplies the architecture, one the size, one the behaviour.

What it is

The problem it was pointed at

Submitted to arXiv on 12 June 2017 and presented at the 31st Conference on Neural Information Processing Systems in December 2017, the paper attacks a single structural property of every sequence model that preceded it. A recurrent network reads a sequence one position at a time, and each step needs the previous step's state. The paper's statement of the cost:

> This inherently sequential nature precludes parallelization within training > examples, which becomes critical at longer sequence lengths, as memory > constraints limit batching across examples.

The contemporary alternatives — ConvS2S, ByteNet, the Extended Neural GPU — attacked this with convolutions, and the paper's objection to them is precise and is the hinge of the whole argument:

> In these models, the number of operations required to relate signals from two > arbitrary input or output positions grows in the distance between positions, > linearly for ConvS2S and logarithmically for ByteNet.

The Transformer's answer: "In the Transformer this is reduced to a constant number of operations, albeit at the cost of reduced effective resolution due to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention."

The abstract states the move in one clause that became the field's slogan:

> We propose a new simple network architecture, the Transformer, based solely on > attention mechanisms, dispensing with recurrence and convolutions entirely.

The trade, printed in a table

Section 4, Why Self-Attention, is where a 2026 reader should spend their time, because the paper does something there that most architecture papers do not: it states the conditions under which its own choice is the right one, in a table, with the exponents visible.

| Layer Type | Complexity per Layer | Sequential Operations | Maximum Path Length | |---|---|---|---| | Self-Attention | O(n²·d) | O(1) | O(1) | | Recurrent | O(n·d²) | O(n) | O(n) | | Convolutional | O(k·n·d²) | O(1) | O(log_k(n)) | | Self-Attention (restricted) | O(r·n·d) | O(1) | O(n/r) |

n is the sequence length, d the model's representation width. The paper is explicit that self-attention wins on the first row's arithmetic conditionally:

> self-attention layers are faster than recurrent layers when the sequence length > n is smaller than the representation dimensionality d, which is most often the > case with sentence representations used by state-of-the-art models in machine > translations, such as word-piece and byte-pair representations.

At the time of writing, d was 512 and n was one sentence. The condition held by a comfortable margin, and the paper says so in the same breath as the claim. Section 3 grades what happened when it stopped holding, and it is the single most useful thing in this entry, because every long-context engineering result of the last five years is a consequence of that one inequality reversing.

The paper also prints its own escape hatch, one row lower in the same table:

> To improve computational performance for tasks involving very long sequences, > self-attention could be restricted to considering only a neighborhood of size r > in the input sequence centered around the respective output position. […] We > plan to investigate this approach further in future work.

The machine

Six encoder layers and six decoder layers, d_model = 512, h = 8 attention heads with d_k = d_v = d_model/h = 64 — "due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality." The rationale for using more than one head is stated a priori rather than measured: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."

Because attention is order-blind, position is injected separately, by a sinusoid: PE(pos,2i) = sin(pos/10000^(2i/d_model)). The paper's stated reason for preferring it over a learned embedding is a forecast, and section 3 grades it:

> We chose the sinusoidal version because it may allow the model to extrapolate > to sequence lengths longer than the ones encountered during training.

Training batches contained "approximately 25000 source tokens and 25000 target tokens" of sentence pairs. The hardware is the part a 2026 reader should sit with. "We trained our models on one machine with 8 NVIDIA P100 GPUs." The base model: 100,000 steps, twelve hours. The big model: 300,000 steps, 3.5 days, at 2.3×10¹⁹ FLOPs. It scored 28.4 BLEU on WMT 2014 English-to-German, beating the previous best including ensembles by over 2 BLEU, and 41.8 BLEU on English-to-French, a new single-model state of the art — "a small fraction of the training costs of the best models from the literature."

One machine, eight cards, a long weekend. The architecture that the entire subsequent build-out rests on was demonstrated for what would now be a rounding error on a single day's depreciation at one datacentre. alexnet-2012 records the same shape one turn earlier — two gaming GPUs — and the pairing is the point: the two most consequential architecture results of the era were both obtained on hardware a university lab could buy, and neither of them predicted, or needed, what came after.

Why a reading would cite it

proposals.md names the occasion as "an architectural claim or a context-length claim is at issue." Both are live, and the entry earns its place by making a reading say which layer of a claim it is reporting.

First: context length, where the advertised number and the working number are different quantities. The readings meet a context figure in nearly every release. DeepSeek-V4, released 24 April 2026, ships both variants at one million tokens; Llama 4 Scout and Gemini 3 Pro are widely reported at ten million. The corrective is a benchmark result, not an opinion. NVIDIA's RULER (Hsieh et al., arXiv, 9 April 2024) evaluated 17 long-context models across 13 tasks and found that "almost all models exhibit large performance drops as the context length increases", and states the gap flatly:

> While these models all claim context sizes of 32K tokens or greater, only half > of them can maintain satisfactory performance at the length of 32K.

The mechanism behind that gap is what this entry supplies. A context window is a capacity, set by what the position machinery and the KV cache will tolerate. It is not a competence, and the paper's own Table 1 says why nobody should have expected it to be: attention gives every position a path to every other position in O(1) steps, which is the property that makes long context possible, and it costs O(n²·d) per layer, which is the property that makes long context expensive and forces every production system to approximate it. A reading that reports "a 1M-token context" as a capability has reported a specification. The finding is what the model does at 1M, and that requires a test somebody ran.

Second: architectural claims, where "post-transformer" and "still a transformer" are both usually true and neither is informative. Nine years on, attention has been attacked continuously — state-space models, linear attention, Mamba and its successors — and the outcome is not replacement but absorption: hybrid designs interleave cheap recurrent or linear layers with a smaller number of full attention layers, and every frontier model whose architecture its builder has published still contains attention. So the base rate this entry supplies is specific: in nine years, no architecture has displaced attention at the frontier, and the successful modifications have all been to its cost rather than to its function.

The sharpest illustration is that the paper's own escape hatch is what shipped. DeepSeek-V4's technical report (arXiv, 26 April 2026) describes "a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency", and reports that "in the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2." That is Table 1's fourth row — restricted self-attention, O(r·n·d) — arriving in production nine years after the paper said "we plan to investigate this approach further in future work" and did not. Note the comparison base carefully: 27% and 10% are against DeepSeek's own previous model, not against dense attention, and they are the vendor's own figures. A reading citing them should say both things.

Third: compute and infrastructure, where this entry explains a shape but not a magnitude. The 16 August 2026 readings are carrying roughly $70bn of off-balance-sheet chip financing and an Nvidia guarantee that moved by $130bn in a week. Serving a transformer is memory-bound in a way the 2017 paper never discusses, because it never generated a long sequence: every token attends to every previous token, so every previous token's keys and values must be held in accelerator memory for the whole generation. That structure — the KV cache — is where a great deal of the physical demand actually sits, and it is the quantity DeepSeek's architecture work is aimed at. The entry supports the mechanism and nothing about the money. I cannot trace a dollar of the financing stories to attention specifically, and a reading should not pretend the chain is short: "attention is quadratic, therefore the capex is justified/unjustified" is two inferences too many and belongs to neither pole.

Fourth, and quieter: what the paper's publication says about the field's disposition, as a dated baseline. Google published the architecture openly in June 2017, under no licence restriction, and every subsequent competitor built on it — including the ones that took the lead. All eight authors subsequently left the company. Whatever a 2026 reading concludes about open weights and closed APIs on the concentration lens, the single most valuable artifact of the decade was given away by the organisation that produced it, nine years ago, and that is a fact about how far the default has moved rather than an argument about where it should sit. neuromancer-1984 and culture-banks-1987 hold the two poles of that axis; this entry supplies one dated point on it and no direction.

What this entry does not support. It is not evidence about any 2026 model's capability, any lab's prospects, or whether any of this is good. It places no score and moves no needle. It does not say attention is the right architecture or the wrong one; it says the claim "we have moved beyond the Transformer" and the claim "it's all still just the 2017 paper" are both cheap, and that the informative question is always which specific property — the mechanism, the complexity, the encoder-decoder shape, the position machinery — is being asserted to have changed. And it makes no claim whatsoever about the architecture of the closed frontier models, whose builders do not publish it.

What it got right, and what it got wrong

Claim 1 — attention alone suffices; recurrence and convolution can be dispensed with entirely. Made 12 June 2017. Right, and more comprehensively than any comparable claim in the field.

Nine years on this is as close to unanimous as the field gets. Wikipedia records the paper at more than 250,000 citations as of 2026, among the top ten most-cited papers of the 21st century — a secondary figure, and cited as one, but no plausible correction changes the conclusion. Every model in every reading this project has taken is a descendant. The mechanism scaled across four orders of magnitude of compute, transferred to images, audio, video, protein structure and robot control, and absorbed each of its challengers rather than yielding to them.

One qualification, and it is not small: the architecture in the paper is not the architecture anyone runs. The paper describes a six-layer encoder feeding a six-layer decoder, built for translation, where the whole source sentence is available before the first output token. The field then split that design in half and used the halves separately — encoder-only for representation (BERT and its line), decoder-only for generation (the GPT line and, so far as their builders disclose, essentially everything generative since). The T in GPT refers to the mechanism, not the model. A reading citing this entry for "the architecture every model descends from" should mean the attention mechanism, the residual-plus- layer-norm block and the position-wise feed-forward sublayer, all of which survive unchanged; it should not mean the network in Figure 1, which nothing runs. This is alexnet-2012's fifth category — correct at the time and obsolete since without ever having been wrong.

Claim 2 — the O(n²·d) trade is acceptable because n < d. Made 12 June 2017 with the condition stated. The condition failed. Honest, conditional, and the source of nine years of engineering.

The paper's own words: self-attention wins "when the sequence length n is smaller than the representation dimensionality d, which is most often the case with sentence representations." With d_model = 512 and n a sentence, that was true.

By 2026 n is one million. d has grown, but nothing like as fast — model widths at the frontier remain in the low thousands, and whatever the exact figure for any particular 2026 model (I did not verify one), n ≫ d now by orders of magnitude. The inequality the paper's central trade depends on has reversed, and the entire efficient-attention literature — sparse patterns, linear attention, FlashAttention, sliding windows, KV compression, the SSM hybrids — exists to cope with it.

The grade is favourable to the paper and the reason matters. It did not overclaim. It printed the condition, printed the alternative row of the table, and named the case where the alternative would be needed. When a 2026 reading meets an architecture paper, this is the standard to hold it to: not "was the choice right" but "did the paper state what its choice depends on." Most do not. The error mode this claim illustrates is not deception but scope — a design validated in a regime, deployed later in a regime nobody had tested, by people who did not re-read the condition.

Claim 3 — restricted, local attention will be needed for very long sequences, and we plan to investigate it. Made 12 June 2017. Right about the need, and it took nine years and other people.

Table 1's fourth row and the accompanying sentence are correct, and the authors did not pursue them. Restricted attention arrived through a long line of other groups' work and is now standard: the 2026 DeepSeek-V4 report's hybrid CSA/HCA scheme, with its reported 10% KV cache at 1M context, is a direct realisation of O(r·n·d).

The transferable lesson is the interval. A correct, published, explicitly flagged research direction from the authors of the most-cited paper in the field took roughly nine years to become production infrastructure. sparse-moe-2017 graded an interval of three and a half years for the same kind of gap and called it the entry's most transferable lesson; this is a second instance and a longer one. A 2026 claim that some architectural idea is "solved in principle" should be read against a base rate measured in years.

Claim 4 — sinusoidal position encodings may let the model extrapolate to longer sequences than it saw in training. Made 12 June 2017. Refuted in print 27 August 2021. Wrong, and instructively so.

The paper: "We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training."

Press, Smith and Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv, 27 August 2021), quote that hedge back and test it. Their finding: "we find that transformer language models (LMs) that use sinusoidal position embeddings have very weak extrapolation abilities", and for a model trained at length L, "this model cannot extrapolate to more than a few dozen tokens beyond L." Perplexity improves for about twenty tokens past the training length, plateaus, and then degrades. Their opening frames the whole episode: "Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training?" Four years, unanswered.

What makes this the most instructive wrong claim in the entry is the structure of the error, which the paper's own ablation exposes. Table 3 row (E) reports swapping the sinusoids for learned positional embeddings: perplexity 4.92 against the base model's 4.92, BLEU 25.7 against 25.8. The paper says so — "we found that the two versions produced nearly identical results." So the one experiment they ran said the choice did not matter, and the reason they gave for making it anyway was an untested a-priori argument that turned out to be false. A design defended by a plausible mechanism rather than by a measurement, in a paper that had already measured that the design made no difference. This is the same failure sparse-moe-2017 graded as its claim 2, where routing to k>1 experts was argued necessary from gradient flow and later shown unnecessary by ablation. Two of the most influential papers in the field, five months apart, sharing an author, each carrying one wrong claim of exactly this shape. When a 2026 reading meets an architectural choice defended from first principles rather than from an ablation, this is the precedent for what that defence is worth.

The claim was hedged with "may", and the entry records that. It was still the stated reason for the choice, and it was still wrong.

Claim 5 — individual attention heads learn different tasks, many related to syntactic and semantic structure. Made 12 June 2017. Half right, and the multi-head rationale was substantially deflated.

The paper: "Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences." The rationale for h=8 was that "with a single attention head, averaging inhibits this."

Two papers two days apart in May 2019 graded it, and they agree. Voita, Talbot, Moiseev, Sennrich and Titov (23 May 2019) confirm the specialization — "the most important and confident heads play consistent and often linguistically-interpretable roles", and "specialized heads are last to be pruned" — and then report the deflation: "on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads results in a drop of only 0.15 BLEU." Michel, Levy and Neubig (25 May 2019) put it more bluntly still: "even if models have been trained using multiple heads, in practice, a large percentage of attention heads can be removed at test time without significantly impacting performance. In fact, some layers can even be reduced to a single head."

So: specialization is real and most of the heads are redundant anyway. The observation was correct; the inference from it — that many heads are therefore necessary — did not survive contact with an ablation. Note the rhyme with claim 4 and with sparse-moe-2017's claim 2 once more: three separate arguments of the form "more than one of these is required, because averaging/gradients", all three partially wrong, all three unfalsified for years because nobody ran the cheap experiment.

Claim 6 — "Making generation less sequential is another research goal of ours." Made 12 June 2017. Still open in 2026.

The Transformer removed sequential computation from training and left it untouched in generation: a decoder-only model still emits one token at a time, each conditioned on the last. Nine years later that is still how the frontier generates. Diffusion language models — Inception Labs' Mercury, DeepMind's Gemini Diffusion — are commercially available and generate in parallel at reported speeds above 1,000 tokens per second, but on the evidence I found they remain competitive mainly on code and short-form output and behind autoregressive models on long-form coherence and complex reasoning. My sources for that 2026 status are secondary technical write-ups rather than benchmark papers, and the reading of "still behind" should be treated as provisional. What is not provisional is that the stated goal is nine years old and the frontier still decodes serially, which is worth carrying as a base rate on how long a named, funded, obvious research goal can stay open.

Commonly misused as

Not required for an idea, but the misreadings here are load-bearing for how 2026 releases get reported.

"The attention weights show you what the model was looking at."

The paper planted this — "As side benefit, self-attention could yield more interpretable models" — and it is the most durable overreach in the entry. It was litigated directly and the literature's own titles record the outcome: **Jain and Wallace, Attention is not Explanation (2019)**, showing that very different attention distributions can produce equivalent predictions, so no particular distribution can be claimed as the explanation; and **Wiegreffe and Pinter, Attention is not not Explanation (EMNLP 2019), arguing the first paper did not establish that attention never explains, only that the claim depends on a definition of explanation the first paper had not pinned down. The honest position is the one the exchange leaves you with: an attention map is a description of where weight went, and converting that into a claim about what the model used requires an argument that is usually not made.** A reading meeting an attention heatmap offered as evidence of a model's reasoning should treat it as a visualization, not a measurement.

"A 1M-token context window means the model reads a million tokens."

It means a million tokens can be supplied without the system refusing. RULER's finding is that even the far more modest 32K claims were half-honoured in 2024. The window is a specification of the plumbing; effective performance is an empirical quantity that degrades with length and degrades faster on tasks needing several facts at once than on tasks needing one. Advertised context and working context are different numbers and the gap widens as the advertised number grows. A reading should quote a test or quote the vendor and say which it is doing.

"Transformers have been replaced." / "Nothing has changed since 2017."

Both errors are common and they are made about the same systems. The attention mechanism is intact and universal; almost everything around it has been rebuilt — decoder-only rather than encoder-decoder, rotary rather than sinusoidal position, sparse mixture-of-experts rather than dense feed-forward, compressed and sparse attention rather than the full quadratic form, and normalization, activation and optimizer all changed. "Is it still a transformer" is not a question with information in it. The question with information is which of those specific components a claim asserts has changed, and what was measured after it did.

"GPT-4 is a transformer, therefore it works the way the paper describes."

The paper's model is 213 million parameters at its largest, six layers deep, trained for 3.5 days on eight P100s to translate sentences. Whatever the mechanism's continuity, no property demonstrated in that paper transfers automatically to a system five orders of magnitude larger, differently trained, and post-trained on human preferences — which is rlhf-christiano-2017's territory and is not in canon/ yet. The architecture is inherited; the behaviour is not.

The one this canon has to watch in itself

"It descends from the Transformer" is true of everything, and therefore explains nothing. This is the specific failure mode of a canon entry this influential: it is available as a citation on every occasion, it always sounds knowledgeable, and attaching it to a finding adds no information at all. LENSES.md asks for what happened, dated, and the discipline this entry imposes on itself is narrow: it should be cited when a specific claim is at issue — a context length, a complexity, an interpretability inference, a "post-transformer" architecture claim, a specialization story — and never as background colour. Every case in section 2 names the exact claim it disciplines. If a reading cannot name one, it should not cite this entry.

Sources

Primary, read directly:

• **Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, Attention Is All You Need, arXiv:1706.03762, submitted 12 June 2017; NeurIPS 2017.** Read as the ar5iv HTML rendering. Source for the abstract; the sequential-nature and parallelization sentences; the ConvS2S/ByteNet distance-growth passage and the constant-operations reply; Table 1 in full; section 4's three desiderata and the n < d condition; the restricted-attention passage and "we plan to investigate this approach further in future work"; section 3.2.2's multi-head rationale and the h=8, d_k=d_v=64 configuration; section 3.5's positional encoding formula, the linear-function-of-offset hypothesis and the extrapolation sentence; Table 3 row (E)'s 4.92/4.92 perplexity and 25.7/25.8 BLEU; the 8×P100 / 12 hours / 3.5 days / 100k and 300k steps training description; the 28.4 and 41.8 BLEU results and 2.3×10¹⁹ FLOPs; the interpretability side-benefit sentence and the attention-head specialization claim; and the conclusion including the future-work sentences about other modalities and less sequential generation. The bibliography was read for descends_from: reference [32] is sparse-moe-2017, quoted above with its in-text citation; references [2], [9] and [13] are Bahdanau et al., Gehring et al. and Hochreiter & Schmidhuber. Krizhevsky et al. 2012 is not cited, so no link to alexnet-2012 is drawn.
• **Ofir Press, Noah A. Smith, Mike Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, arXiv:2108.12409, submitted 27 August 2021 (v1), revised 22 April 2022.** Abstract and body read. Source for the whole of claim 4's grading: the opening "a fundamental question has yet to be answered", the "very weak extrapolation abilities" finding, the "cannot extrapolate to more than a few dozen tokens beyond L" result, and the perplexity-plateau-then-degrade pattern at k=20 to k=50.
• **Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov, Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned, arXiv:1905.09418, 23 May 2019.** Abstract read. Source for the linguistically-interpretable-roles finding, "specialized heads are last to be pruned", and the 38-of-48-heads / 0.15 BLEU result on English-Russian WMT.
• **Paul Michel, Omer Levy, Graham Neubig, Are Sixteen Heads Really Better than One?, arXiv:1905.10650, 25 May 2019; NeurIPS 2019.** Abstract read. Source for the removal-at-test-time finding and "some layers can even be reduced to a single head."
• **Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg (NVIDIA), RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv:2404.06654, 9 April 2024, revised 6 August 2024.** Abstract read. Source for the 17-models/13-tasks scope, "almost all models exhibit large performance drops as the context length increases", and the 32K claim-versus-performance sentence.
• **DeepSeek-AI et al., DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv:2606.19348, 26 April 2026.** Abstract and technical summary read. Source for the hybrid CSA/HCA description and the "only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2" figure. Both figures are the vendor's own and are relative to DeepSeek-V3.2, not to dense attention; the entry says so where they appear. The 1M context and 1.6T/49B and 284B/13B parameter figures corroborate the DeepSeek model card already read for sparse-moe-2017, which gives Flash as 285B — that discrepancy is carried in that entry and not re-adjudicated here.
• **Sarthak Jain and Byron C. Wallace, Attention is not Explanation (NAACL 2019) and Sarah Wiegreffe and Yuval Pinter, Attention is not not Explanation, arXiv:1908.04626 (EMNLP 2019).** Findings read via search results and the ACL Anthology listing rather than the full papers; the characterizations in section 4 are at the level of each paper's central claim and should be checked against the originals before any narrower use.

Secondary, and marked as such:

• **Wikipedia, Attention Is All You Need.** Source for the >250,000 citation count "as of 2026" and the top-ten-most-cited-of-the-century placement, the NeurIPS December 2017 venue, the randomized author order and equal-contribution footnote, the statement that all eight authors subsequently left Google, and the Beatles origin of the title. Bibliographic and biographical facts only; nothing technical in this entry rests on it. The citation figure in particular is a secondary number of a moving quantity.
• The 2026 hybrid-architecture picture — that state-space and linear-attention work has produced interleaved hybrids rather than replacements, and that attention persists in every architecture-disclosing frontier model. Assembled from search results over 2026 survey and vendor material, not from a single primary source. The direction is well attested; no specific model-by-model claim should be drawn from this entry.
• Diffusion language model status in 2026 — Mercury (Inception Labs) and Gemini Diffusion (DeepMind) as shipped commercial products, >1,000 tokens/second, competitive on code and short-form and behind on long-form coherence and complex reasoning. From secondary technical write-ups. Claim 6's grade of "still open" rests on the uncontroversial part (the frontier still decodes autoregressively); the "behind on reasoning" characterization is provisional.
• The 10M-token context figures for Llama 4 Scout and Gemini 3 Pro. From search-result summaries, not from either vendor's own documentation. Used only as an order of magnitude.

Not obtained:

• The NeurIPS 2017 reviews and decision record for the paper. As with sparse-moe-2017, contemporaneous judgment is the most valuable missing source for an entry that grades in hindsight — what the field thought of this in September 2017, before it was obviously right, is exactly what a canon should prefer and exactly what is hardest to get.
• Bahdanau, Cho and Bengio (2014), the origin of the attention mechanism this paper's title makes a claim about. Only the reference entry was obtained. It is the most conspicuous gap in canon/ that this entry exposes, and it is not currently in proposals.md under any id.
• A verified d_model for any 2026 frontier model, without which claim 2's inequality reversal is stated qualitatively rather than with two numbers. The reversal itself is not in doubt at n = 10⁶; the precise margin is not established here.
• Any independent benchmark of long-context performance at 1M tokens across the 2026 frontier. The most promising-looking aggregation found (a March 2026 comparison of MRCR v2 scores) discloses that it is 100% AI-generated and states that the scores it collects "cannot be directly compared" across sources; a search summary of the same page returned numbers that do not appear on it. It is not cited and none of its figures appear in this entry. RULER is used instead, and it is two years old — a 2024 result about 32K, carried because it is a real benchmark paper, not because it is current. This is the entry's weakest evidential point and a reading leaning on section 2's first case should look for something newer.
• Any disclosure of whether the closed frontier models — Claude, GPT-5.x, Gemini — use the architecture described here, or any variant of it. Their vendors do not publish architecture. This entry makes no claim either way, on the same discipline sparse-moe-2017 adopted.