Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

idea is right, and the id was filed correctly. This is the canon's third idea, after shannon-chess-1950 and dijkstra-1959, and it is the same shape of thing those two defined: a framing that later work is built out of rather than argued about, where the citation occasion is not "someone is wrong about this" but "the thing you are watching descends from this, and the descent explains its shape." Nothing here is a limit — the paper proves no impossibility, and its central claim is that an assumed limit was not one. Nothing here is a moment: there was no adjudicated public event, no withheld test set and no rival field showing up to be beaten, which is the test alexnet-2012 and rosenblatt-perceptron-1958 settled between them. It was a poster at ICLR 2017.

It is also not a prediction, and section 3 grades its forecasts anyway. shannon-chess-1950 set that precedent for exactly this reason: "a canon that only grades entries filed under prediction will systematically miss the forecasts that arrive embedded in engineering papers — which is where most of them actually live." This paper carries at least five dated, falsifiable technical claims, two of which were refuted in print by later papers with an overlapping author, and the grading is where a good deal of the entry's value sits.

descends_from holds one id, and it is documented by citation rather than inferred from subject matter. The paper's bibliography carries Krizhevsky et al. 2012, and the in-text citation is in the introduction's opening move — "This has been shown in domains such as text …, images (Krizhevsky et al. 2012; Le et al. 2012), and audio …" — offered as evidence for the two sentences the paper opens with:

> Exploiting scale in both training data and model size has been central to the > success of deep learning. When datasets are sufficiently large, increasing the > capacity (number of parameters) of neural networks can give much better > prediction accuracy.

That is the scaling thesis, asserted as settled, with AlexNet cited as one of three proofs. alexnet-2012 is the entry that dates the origin of that thesis to a sentence — "our results can be improved simply by waiting for faster GPUs and bigger datasets to become available" — and this paper's entire contribution is an attack on the price of the bet that sentence opened, not on the bet. The descent is both documented and explanatory, which is the standard alexnet-2012 set when it declined to link things it merely resembled. Since alexnet-2012 descends from rosenblatt-perceptron-1958, the chain to the perceptron exists transitively and should not be drawn directly: this paper does not cite Rosenblatt.

The proximate ancestors are three, and none of them is in canon/:

• Jacobs, Jordan, Nowlan and Hinton (1991), Adaptive Mixtures of Local Experts — the actual origin of everything the 2017 paper's title refers to, cited in its first sentence about mixtures of experts, described in section 1 below in detail because the 2017 paper is unintelligible without it. Not proposed under any id. proposals.md names it in this entry's own description and gives it no file.
• The Transformer — Vaswani et al., arXiv 12 June 2017, on which Shazeer is second author. Proposed as transformer-2017 and unwritten. Section 3 grades what its arrival five months later did to this paper.
• Backpropagation — Rumelhart, Hinton and Williams, 1986, which alexnet-2012 already recorded as "left out at the cap" of the proposal list. The 1991 paper's opening sentence is about backpropagation's failure mode, so the absence bites here too.

scaling-laws-2020, also proposed and unwritten, is the entry that would quantify what this one makes cheap, and the two should eventually be read together.

What it is

The 1991 original, whose problem was not the 2017 problem

The mixture-of-experts idea is a full generation older than the paper this entry is named for, and the gap matters more than the resemblance. Jacobs, Jordan, Nowlan and Hinton published Adaptive Mixtures of Local Experts in Neural Computation 3, 79–87 (1991) — received 27 July 1990, accepted 1 November 1990, communicated by Jeffrey Elman, with a footnote recording that "This idea was first presented by Jacobs and Hinton at the Connectionist Summer School in Pittsburgh in 1988." Jacobs and Jordan were at MIT's Department of Brain and Cognitive Sciences; Nowlan and Hinton at Toronto.

Their motivating problem was interference, not capacity. The paper's first sentence:

> If backpropagation is used to train a single, multilayer network to perform > different subtasks on different occasions, there will generally be strong > interference effects that lead to slow learning and poor generalization.

The proposed fix is "several different 'expert' networks plus a gating network that decides which of the experts should be used for each training case." The gating network already has the form it still has in 2026 — the paper's Figure 1 caption specifies "normalized outputs p_j = exp(x_j)/Σ_i exp(x_i)", which is a softmax over gate logits.

The paper's real technical contribution is a change of loss function, and it is a change from cooperation to competition. The prior formulation compared the target against a blended output, which meant "each local expert must make its output cancel the residual error that is left by the combined effects of all the other experts" — a coupling that "causes them to cooperate nicely, but tends to lead to solutions in which many experts are used for each case." Their remedy was to make the gate choose "a stochastic decision about which single expert to use on each occasion" so that each expert "is required to produce the whole of the output vector rather than a residual", and "the local experts are encouraged to compete rather than cooperate."

The demonstration was tiny, and its result is not what the retellings imply. The task was a speaker-independent four-class vowel discrimination problem — the first and second formants of [i], [I], [a] and [A] from 75 speakers, out of Peterson and Barney's 1952 data, trained on the first 50 speakers and tested on the remaining 25. Mixtures of 4 or 8 experts were compared against ordinary backpropagation networks with 6 or 12 hidden units. The results table:

| System | Train % correct | Test % correct | Avg. epochs | SD | |---|---|---|---|---| | 4 Experts | 88 | 90 | 1124 | 23 | | 8 Experts | 88 | 90 | 1083 | 12 | | BP 6 Hid | 88 | 90 | 2209 | 83 | | BP 12 Hid | 88 | 90 | 2435 | 124 |

Every system scores identically. The paper says so and explains why — "the set of misclassified examples is not sensitive to small changes in the decision surfaces" — and the entire measured advantage is convergence speed, "requiring only about half as many epochs on average." The mixture of experts arrived as a training-time optimization on a four-way vowel task, not as a capacity mechanism, and it did not improve accuracy at all.

And the paper reports expert collapse, as a virtue. In the caption to Figure 2: "Although the mixture in this case contained 4 experts, one of these experts made no significant contribution to the final mixture since its mixing proportion p was effectively 0 for all cases … the system tends to use as few experts as it can to solve a problem." The closing sentence of the body is blunter: "This solution is typical — in all simulations with mixtures of 4 or 8 experts all but 2 or 3 experts had mixing proportions that were effectively 0 for all cases."

That is the load-balancing problem, observed in 1991, named as a feature. It is the single hardest engineering obstacle to everything built on this idea since, and section 3 grades three successive attempts to solve it. A canon reader should hold the shape of that: the defect that would take thirty-three years of engineering to get right was visible in the original paper and was written up as economy.

What the 2017 paper actually did

Shazeer et al. submitted to arXiv on 23 January 2017 and presented as a poster at the 5th International Conference on Learning Representations, Toulon, France, 24–26 April 2017. The paper takes the 1991 machinery and points it at a different problem, stated in its own first line of abstract:

> The capacity of a neural network to absorb information is limited by its number > of parameters.

The move that makes the entry worth having is the one the abstract makes next:

> Conditional computation, where parts of the network are active on a per-example > basis, has been proposed in theory as a way of dramatically increasing model > capacity without a proportional increase in computation.

That sentence is the idea. Before it, "how big is this model" had one answer, because every parameter participated in every forward pass, so parameter count was simultaneously a statement about memory, about training cost, and about the price of a single prediction. After it, those come apart. A model has a total parameter count, which is what it knows and what you must hold in memory, and an active parameter count, which is what runs for any given token and therefore what it costs. The 2026 readings meet this every time a release is announced, and it is the reason a parameter count now arrives as two numbers.

The paper's specific contributions, against the 1991 baseline:

• Sparsity by top-k, not by sampling. The gate is G(x) = Softmax(KeepTopK(H(x), k)) — everything outside the top k is set to −∞ so it softmaxes to zero, and its expert is never evaluated. In the reported language-modelling models "4 experts were active per input" out of as many as 131,072.
• Noisy gating. H(x)ᵢ = (x·W_g)ᵢ + StandardNormal()·Softplus((x·W_noise)ᵢ) — tunable per-expert noise added before the top-k cut, to help spread load.
• An explicit balancing loss. Rather than let the system "use as few experts as it can", they add a term "equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor." This is the 1991 collapse observation, inverted into a penalty.
• A distributed-batching answer to the shrinking batch problem. Their own statement of the difficulty: "If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb/n ≪ b examples." Sparsity destroys the batch sizes GPUs need. Their fix is to run many data-parallel model replicas synchronously and combine their per-expert traffic, so each expert sees a batch gathered across the whole cluster.

The paper is candid that these are the reasons the idea had not worked, and lists the barriers plainly: GPUs are "much faster at arithmetic than at branching"; "large batch sizes are critical for performance"; network bandwidth limits how fast parameters can be moved; balancing losses cost quality; and enormous capacity needs enormous data to train against.

The results, and the machine

Applied "convolutionally between stacked LSTM layers":

• 1 Billion Word Benchmark — 28.0 perplexity from a 4.3-billion-parameter MoE model, against 30.6 for the best previously published result.
• 100 Billion Word Google News corpus — a 65,536-expert model (68 billion parameters) reaching 39% lower perplexity than the baseline; the largest model reported ran 131,072 experts and 137.7 billion parameters.
• WMT'14 En→Fr — 40.56 BLEU from a 2,048-expert model with 8.7B parameters and 85M ops per timestep. WMT'14 En→De — 26.03 BLEU.

The hardware is the part a 2026 reader should sit with: "We trained our models using TensorFlow on clusters containing 16-32 Tesla K40 GPUs", with translation training "3 days/64 k40s", at a reported "computation efficiency ranged from 0.74-0.90 TFLOPS/GPU." A 137-billion-parameter model existed in January 2017, on four-year-old server cards, four and a half years before anything of comparable total size was a product. What it lacked was not scale. It was a body.

Why a reading would cite it

The occasion is the one proposals.md names, and it is live and dated. On 24 April 2026 DeepSeek released V4 under the MIT licence, and its own model card — publication date 27 April 2026 — states the model size in two numbers, in one cell:

> Pro: 1.6T parameters, of which 49B are activated for each token > > Flash: 285B parameters, of which 13B are activated for each token

A reading that meets that line and reports "a 1.6-trillion-parameter model" has not misquoted anything and has still lost the finding. The two numbers measure different physical quantities and imply different conclusions on different lenses, and this entry exists to make a reading say which one it means.

• 49B is the compute number. It sets the floating-point work per token, and therefore the marginal cost of generation, the price of the API call, and — via tokens consumed in training — the order of the training bill. On this number V4-Pro is a mid-sized model.
• 1.6T is the memory number. Every parameter must be resident somewhere the accelerators can reach, because the router may call any of them for the next token. On this number V4-Pro is among the largest artifacts ever shipped. The HuggingFace card gives the precision as "FP4 + FP8 Mixed", with "MoE expert parameters use FP4 precision; most other parameters use FP8" — at four bits for the bulk of 1.6T parameters, that is on the order of 800 GB of weights before any KV cache, which is roughly six to ten current datacentre accelerators. That arithmetic is mine; the card does not state a weight size, and I did not obtain one.

The sharpest available demonstration that neither number is "the size" is a single vendor's own product line. Meta's Llama 4 Scout and Maverick both activate 17B parameters per token; Scout totals 109B across 16 experts and Maverick totals 400B across 128. Identical compute per token, a 3.7× spread in memory, and different capability. No single scalar orders those two models. The industry has quietly conceded the point by encoding it in model names — Alibaba ships Qwen3-235B-A22B, where "A22B" means 22B activated of 235B total, 8 of 128 experts per token.

This is the entry that makes the concentration lens read an open-weight release correctly, and the reading cuts both ways. DeepSeek-V4-Pro is MIT-licensed: freely redistributable, modifiable, commercially usable, with the weights on HuggingFace. It is also, for almost everyone, unrunnable — the licence is open and the hardware requirement is a rack. Compare gpt-oss-120b, Apache 2.0, 117B total and 5.1B active, which OpenAI states "run[s] on a single 80GB GPU", and gpt-oss-20b at 21B/3.6B running "within 16GB of memory." Same architecture family, same licence posture, and one of them a person can own while the other is something a person may only rent. A reading that treats "open weights" as a single binary is losing the axis the lens is actually about, and MoE is the mechanism that split it: total parameters set who can possess a frontier model, active parameters set what it costs to serve one. neuromancer-1984 holds the frame for the first reading and culture-banks-1987 for the second, and the honest position is that this architecture pushed the two apart rather than choosing between them.

On the compute and infrastructure lens, the entry is a guard against arithmetic on the wrong number. The 16 August 2026 readings are carrying roughly $70bn of off-balance-sheet chip financing and an Nvidia guarantee that moved by $130bn in a week. Any inference from "the models are getting bigger" to "the compute bill scales with that" has to say which parameter count it means, because since 2017 the two have been free to move in opposite directions — and they have. Serving cost per token and total model size are no longer the same curve, which is why a 1.6T model can undercut a much smaller dense one on price.

On safety, there is a narrower and quieter occasion. The obvious inference from the word "expert" — that a model's dangerous capability lives in some identifiable subset of it, which could be located and removed — is checkable, and someone checked it. A routing analysis of Mixtral 8x7B-Instruct under benign and harmful prompts (arXiv, 22 May 2026) reports that "most experts are shared across both prompt types", that only "a limited subset shows clear group preference", and that the safety signature is "subtle, depth-dependent, and distributed rather than dominated by a fixed set of experts." That is a single-author preprint and should be cited as one, but it is the right shape of evidence and it points the way the architecture would predict: there is no refusal expert to ablate, and none to attack. This lens's own discipline — quiet primary ground over commentary — is what surfaces findings like it.

What this entry does not support. It is not evidence about any 2026 model's capability, any lab's prospects, or any claim about whether large models are good. Nothing in it moves the needle, and it places no score. It does not say that MoE models are better than dense ones, or worse; it says that the single-number comparison people reach for when they want to say either is not available. It is not an argument that scaling continues. And it specifically does not license the inference that a large total parameter count is impressive: section 5 is about how often that inference gets made in both directions.

What it got right, and what it got wrong

Not required for an idea, but this paper made unusually many checkable technical claims, and two of them were refuted in print by later papers carrying one of its own authors — which is the most useful kind of grading available, because nobody can call it hindsight from outside.

Claim 1 — conditional computation works, and gives >1000× capacity at minor cost. Made 23 January 2017. Right, comprehensively, and it needed a substrate the paper did not have.

The abstract's claim is "greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters", and the conclusion's is "This work is the first to demonstrate major wins from conditional computation in deep networks."

Graded over nine years: correct, and if anything conservative about what the idea would carry. The lineage is unbroken and public. GShard (Lepikhin et al., 30 June 2020) moved the sparsely-gated MoE layer onto the Transformer and trained "beyond 600 billion parameters … on 2048 TPU v3 accelerators in 4 days" for 100-language translation — Shazeer is an author. Switch Transformer (Fedus, Zoph and Shazeer, 2021) reached 1.571 trillion parameters. DeepSeek-V3 (December 2024) was "a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token", trained for 2.788M H800 GPU-hours on 14.8T tokens. By 2026 the disclosed open-weight frontier is overwhelmingly this architecture: DeepSeek-V4-Pro at 1.6T/49B, Moonshot's Kimi K2 reported at about 1.04T total and ~32B active across 384 experts, Qwen3 at 235B/22B, Llama 4 at 400B/17B, gpt-oss at 117B/5.1B.

One honest qualification, and it is large. The paper's own demonstration vehicle was already dying as it was published. Its MoE layers were "applied convolutionally between stacked LSTM layers" — and **140 days after this paper hit arXiv, its first author was the second author on Attention Is All You Need (12 June 2017), which retired the recurrent substrate the results were obtained on. The idea survived the transition; the paper's architecture did not, and it took until GShard in 2020 — three and a half years — for the combination that matters to be demonstrated at scale. A reading citing this entry for "the idea was proven in 2017" should say it was proven on hardware and in a network family that were both obsolete within four years, and that the gap between a correct idea and a usable one was measured in years, not months.** That gap is the entry's most transferable lesson and it applies directly to any 2026 architecture claim.

I did not establish what fraction of the Tesla K40's peak throughput the reported "0.74-0.90 TFLOPS/GPU" represents, so this entry does not grade "only minor losses in computational efficiency" numerically. The claim as stated is the authors' own characterization of their own overhead.

Claim 2 — routing to k>1 experts is necessary. Made January 2017. Refuted in print by 2021. Wrong.

The paper's rationale for its top-k gate: "If we choose k>1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network." The Switch Transformer paper states the conjecture it is overturning in as many words — "Shazeer et al. 2017 conjectured that routing to k>1 experts was necessary in order to have non-trivial gradients to the routing functions" — and then routes to exactly one: "Switch routing uses a simplified strategy where we route to only a single expert." It works, it is cheaper, and Shazeer is the third author on the refutation.

This is the most instructive wrong claim in the entry, because of its direction. The error was toward unnecessary complexity, justified by a plausible mathematical argument about gradient flow that turned out not to bind in practice. Nobody measured whether k=1 failed; it was reasoned that it must. When a 2026 reading meets an architectural design defended by an argument from first principles rather than an ablation, this is the precedent for how much that argument is worth.

Claim 3 — the auxiliary importance loss solves load balancing. Made January 2017. Standard for seven years, then displaced. Right problem, temporary fix.

Shazeer's balancing loss with its "hand-tuned scaling factor" was the standard answer through GShard and Switch, and it was answering a real and permanent problem — the same collapse Jacobs et al. had observed in 1991 and called economy. Then DeepSeek-V3, December 2024, "pioneers an auxiliary-loss-free strategy for load balancing", and DeepSeek-V4's model card records that it "retains the DeepSeekMoE framework". The auxiliary loss was itself a source of quality degradation — the paper's own list of barriers had said as much, that loss terms for sparsity control "affect model quality" — and it took seven years for someone to remove it rather than tune it.

This is the category alexnet-2012 isolated as claim 5: correct at the time and obsolete since without ever having been wrong. The canon should keep it distinct from claim 2, which was wrong on arrival.

Claim 4 — "the different experts tend to become highly specialized based on syntax and semantics." Made January 2017. Half right, and the wrong half is the one the name promises.

The paper asserts this and supports it in Appendix E Table 9, with examples of individual experts attaching to particular contexts. Mixtral of Experts (Mistral AI, January 2024) went looking for the same thing at production scale and reported the opposite for the semantic half:

> Surprisingly, we do not observe obvious patterns in the assignment of experts > based on the topic.

Expert distributions over ArXiv papers, biology and philosophy documents came out "very similar" at every layer. What Mistral did find was the syntactic half, intact and strong: "This suggests that the router does exhibit some structured syntactic behavior", with "words such as 'self' in Python and 'Question' in English often get routed through the same expert", indentation tokens "always assigned to the same experts", and consecutive tokens showing "high positional locality".

So: syntax yes, semantics no — and the semantic reading is the one carried by the word "expert," by every diagram with a stethoscope on one box and a gavel on another, and by a good deal of 2026 marketing. Section 5 grades that.

There is a genuine and interesting wrinkle here that a careless reading would flatten. In 1991 the semantic claim was true. The vowel paper's Figure 2 shows "different experts learn to concentrate on one pair of classes or the other", and Figure 3 traces the decision lines of individual experts converging on the [i]/[I] boundary and the [a]/[A] boundary respectively. At four experts on a four-class task where task decomposition was the explicit goal, the experts were experts. The property did not survive the change of purpose. What is being called an expert in 2026 is a 1991 word doing work the 1991 evidence does not support, and the drift happened because the mechanism was retargeted from decomposing a task to storing parameters.

Claim 5 — capacity is what limits a network. Made January 2017. Superseded rather than graded.

The abstract's opening premise — capacity limited by parameter count, so raise parameters — was mainstream in 2017 and was refined three years later by scaling-laws-2020 (Kaplan et al.) and corrected again by Chinchilla, which found parameters had been systematically over-weighted relative to data. That entry is proposed and unwritten, and until it exists this claim's status here is open and pointed at a gap, not graded. A reading should not take this paper as authority on how capability scales; it is authority only on how capacity may be bought.

Commonly misused as

Not required for an idea, but this entry is mostly about a misreading, and the misreadings are load-bearing for how 2026 releases get reported.

"8×7B means 56 billion parameters."

Mixtral 8x7B has 47B total and 13B active — the paper says "each token has access to 47B parameters, but only uses 13B active parameters during inference." The name's arithmetic does not work because only the feed-forward blocks are replicated across experts; attention, embeddings and normalization are shared. Nine billion parameters go missing between the name and the model. This is the most common single error made about MoE models and it is made by people reading the vendor's own name for the product.

"The experts are specialists — there's a medicine one and a code one."

There is not, and section 3 grades the evidence. Two further facts kill the picture even where routing is non-random. Routing is per-token and per-layer, so a single sentence through a 61-layer model is processed by dozens of different expert combinations, re-chosen at every layer for every token; there is no point at which "the medical expert" could be consulted, because the unit of routing is smaller than any unit a human would call a topic. And where specialization is detectable it attaches to "indentation tokens" and the word "self" — that is, to surface form. A reading meeting a vendor claim that a model "routes your query to the right expert" should treat it as a claim about marketing vocabulary until an ablation is shown, and should note that the field's own published routing analyses have looked and mostly not found it.

"Model A has 1.6T parameters and model B has 671B, so A is 2.4× the model."

It is 2.4× the memory. It is not 2.4× the compute (49B against 37B active is 1.3×), not 2.4× the training cost, not 2.4× the serving price, and not any ratio of capability. Comparing total parameters between two MoE models compares storage. Comparing an MoE total against a dense model's parameter count compares nothing at all, because for the dense model the two numbers are the same and for the MoE they are thirty-three times apart. The Llama 4 pair — 109B and 400B at identical 17B active — is the cleanest disproof available that a single number orders these models, and it comes from one vendor's own simultaneous release.

"Sparse means most of the model is wasted."

Every parameter is trained and every parameter is used; none is used for every token. Sparsity here is a statement about per-token activation, not about dead weight. The inverse error is equally common — that MoE is a compression trick, or a way of getting a big model's behaviour cheaply. It is not a compression of anything: the full 1.6T must exist, be stored, be loaded and be paid for.

"MoE made big models cheap."

Cheap per token, expensive per deployment, and this inversion is the whole of the concentration finding. Marginal generation cost fell, which is why frontier- class API pricing fell and why a great many people now have access to capability that would have been unaffordable dense. Fixed deployment cost did not fall — it rose, because the total parameter counts rose faster than the active ones. The architecture simultaneously widened access-by-renting and narrowed access-by-owning. Any reading that reports only one of those two movements has reported half of what happened, and rule 3 makes writing both cases mandatory anyway.

The one this canon has to watch in itself

A large parameter count is an awe number, and it is available to both poles. It can be written up as capability arriving faster than anyone expected, or as capital being consumed at a scale nobody can service, and the same figure serves either. The discipline this entry exists to impose is narrow and mechanical: say which parameter count, say what it measures, and refuse a comparison that holds neither constant. That is LENSES.md's "record the magnitude, never the consequence" applied to the one magnitude the readings meet most often, and it is rule 2's requirement that nothing be dressed as a measurement of something it is not measuring. A needle placed partly because a number was large is a needle placed on a unit nobody checked.

Sources

Primary, read directly:

• **Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, Geoffrey E. Hinton, Adaptive Mixtures of Local Experts, Neural Computation 3, 79–87 (1991).** Read in full as the MIT Press PDF served from cs.toronto.edu/~fritz/absps/. Every quotation of the abstract, the interference sentence, the cooperation-versus-competition argument, equations 1.1–1.3, the Figure 1 gating softmax, the vowel task description, Table 1 and its caption, the Figure 2 and Figure 3 captions, and the closing sentence about mixing proportions "effectively 0 for all cases" is from this text at pages 79–87. Received 27 July 1990, accepted 1 November 1990, communicated by Jeffrey Elman. Its own reference list — Barto 1985, Hampshire and Waibel 1989, Jacobs and Jordan 1991, Jacobs, Jordan and Barto 1991, McLachlan and Basford 1988, Moody and Darken 1989, Nowlan 1990a and 1990b, Peterson and Barney 1952 — contains no entry that is in canon/.
• **Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, arXiv:1701.06538, submitted 23 January 2017; ICLR 2017 (poster), Toulon, France, 24–26 April 2017. Read as the ar5iv HTML rendering. Source for the abstract, the opening two sentences of the introduction, the Krizhevsky in-text citation, the Jacobs/Jordan attribution sentence, the gating equations, the k>1 rationale, the shrinking- batch passage, the importance-loss definition, the specialization claim and Appendix E, all benchmark and parameter figures, the K40 cluster description and TFLOPS/GPU range, and the conclusion. Venue confirmed against the dblp** record conf/iclr/ShazeerMMDLHD17 and the dblp proceedings entry for the 5th ICLR.
• **DeepSeek, DeepSeek V4 Technical Documentation / Model Card, publication date 27 April 2026, updated 24 April 2026, PDF served from fe-static.deepseek.com. Read in full. Source for the release date (24 April 2026), the MIT licence, the 1M context length, the architecture description (DeepSeekMoE framework and Multi-Token Prediction retained from V3; Compressed Sparse Attention and Heavily Compressed Attention; Manifold-Constrained Hyper-Connections; Muon optimizer), and the size cell quoted above. Note a discrepancy worth carrying: the model card gives Flash as 285B; secondary coverage widely reports 284B.** This entry uses the card.
• DeepSeek-V4-Pro model card, HuggingFace. Source for the "FP4 + FP8 Mixed" precision statement and "MoE expert parameters use FP4 precision; most other parameters use FP8." The card does not state a weight-file size; the ~800 GB figure in section 2 is my arithmetic from the parameter count and precision, and is flagged as such where it appears.
• **DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437.** Source for 671B total / 37B activated, the "auxiliary-loss-free strategy for load balancing" phrase, 14.8T training tokens and 2.788M H800 GPU-hours.
• **Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, arXiv:2006.16668, 30 June 2020.** Abstract read. Source for 600B+ parameters, 2048 TPU v3, four days, 100 languages, and the move of sparsely-gated MoE onto the Transformer.
• **William Fedus, Barret Zoph, Noam Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, arXiv:2101.03961.** Source for Switch-C at 1,571 billion parameters, the k=1 routing simplification, the characterization of Shazeer et al. 2017's k>1 conjecture, the bfloat16/selective-float32 stability work, and the 7.5× step-time speedup over T5-Base.
• **Mistral AI, Mixtral of Experts, arXiv:2401.04088.** Source for 47B total / 13B active and the whole of the routing-analysis grading in section 3: the "we do not observe obvious patterns … based on the topic" finding, the structured-syntactic-behaviour sentence, the self/Question/indentation examples, and positional locality.
• **Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, Attention Is All You Need, arXiv:1706.03762, submitted 12 June 2017.** Bibliographic detail only, for the date and Shazeer's authorship position. The paper was not read for this entry; transformer-2017 is where it belongs.
• Qwen3-235B-A22B model card, HuggingFace. Source for 235B total / 22B activated, 128 experts with 8 activated per token, 94 layers, and the meaning of the "A22B" naming convention.
• gpt-oss-120b model card, HuggingFace. Source for 117B total / 5.1B active, Apache 2.0, MXFP4 quantization of the MoE weights, "run on a single 80GB GPU" and gpt-oss-20b at 21B/3.6B "within 16GB of memory." OpenAI's own announcement page returned HTTP 403 and was not read.
• **Md Nurul Absar Siddiky, Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts, arXiv:2605.24270, 22 May 2026. Abstract and findings read. A single-author preprint, cited as such** — source for the distributed safety signature and the suppression-intervention results.

Secondary, and marked as such:

• Llama 4 parameter figures — Scout at 109B total / 17B active / 16 experts, Maverick at 400B total / 17B active / 128 experts. Taken from search results summarizing Meta's announcement and the HuggingFace card rather than from Meta's blog post directly, which was not read.
• Kimi K2 figures — approximately 1.04T total, ~32B active, 384 experts with 8 routed plus 1 shared per token, 15.5T pre-training tokens, MuonClip optimizer. From secondary technical write-ups; Moonshot's own report was not read, and the numbers here are stated as reported.

Not obtained:

• The OpenReview record for the paper (openreview.net/forum?id=B1ckMDqlg), which returned a browser-verification wall. This would have given the ICLR 2017 reviews and the area chair's decision text — that is, what the field thought of the idea at the time, before it was vindicated. That is the single most valuable missing source for this entry, because a canon that grades hindsight should prefer contemporaneous judgment where it can get it.
• Jacobs and Hinton's 1988 Connectionist Summer School presentation, which the 1991 paper's own footnote names as the first airing of the idea. Only the footnote was obtained.
• **Jordan and Jacobs (1994), *Hierarchical Mixtures of Experts and the EM Algorithm***, cited by the 2017 paper alongside Jacobs et al. 1991 and not read here.
• A stated weight-file size for DeepSeek-V4-Pro, and any vendor statement of minimum inference hardware for it. The memory argument in section 2 rests on arithmetic from disclosed parameter counts and disclosed precision, not on a vendor figure.
• Any disclosure of whether the closed frontier models — Claude, GPT-5.x, Gemini — are mixture-of-experts. Their vendors do not publish architecture, widely circulated claims about them are unsourced, and this entry makes no claim either way. Everything asserted above about the 2026 landscape is drawn from models whose builders published the numbers.
• The Tesla K40's peak throughput, without which the 2017 paper's "only minor losses in computational efficiency" cannot be graded numerically. It is left ungraded rather than estimated.