A Mathematical Theory of Communication

Filed correctly, and for once the filing needed no argument at all. limit is where canon/proposals.md put it, limit is what the project's own definition describes — "a result about what cannot be done, carried together with what it is repeatedly made to say instead" — and Theorem 11 is a theorem with a proof, a stated set of hypotheses, and a large territory outside them. This is the canon's fourth entry of the kind, after godel-incompleteness-1931, turing-halting-1936 and perceptrons-1969.

But it is the odd one out among the four, and that is the entire reason to have it. The other three tell you a region is empty. Gödel: there is no complete consistent recursively axiomatised theory of arithmetic. Turing: there is no general decision procedure. Minsky and Papert: a single-layer perceptron cannot compute parity. Each is a wall with nothing on the far side, and each is therefore quotable in exactly one direction — as a reason something is impossible. Shannon's result is a wall with a floor attached. Theorem 11 says two things in one breath: above the rate C nothing works, and below the rate C everything works, to any reliability you care to name. It forbids and it promises, and the promise is the harder half and the half that was actually in doubt.

That two-sidedness is what makes this the canon's reference specimen of an honest limit, and it is the reason the entry exists. When a reading meets a claim of the form "X is impossible, for information-theoretic reasons" — and it meets several a month — the useful question is not whether the claimant has cited Shannon correctly. It is whether their limit has the properties Shannon's has: a stated channel model, a proof, a converse, a matching achievability result, and a measurable gap that engineering can be observed closing. Almost none do. Section 4 is about what happens when the name is borrowed without the properties.

On the id and the title. The id names one result; the title names the whole paper, which is 55 pages in the corrected Bell System Technical Journal reprint and contains twenty-three numbered theorems, seven appendices, the definition of entropy, the word bit, the first published Hamming code, the sampling-theorem apparatus, a Markov model of English, and the germ of rate–distortion theory. This entry is about the capacity result — Theorem 11 for the discrete channel, Theorem 17 for the band-limited Gaussian one, and the object that popular usage calls "the Shannon limit." The rest of the paper is described only where it bears on that. A reading that needs Shannon's entropy for its own sake, rather than as the input to a capacity calculation, is using a different part of this file's subject and should say so.

descends_from is empty, and here it is a checked fact rather than a gap. The 1948 paper carries no bibliography at all — every reference is a footnote — and I read them. Footnotes 1 and 2, on the first page, are Nyquist ("Certain Factors Affecting Telegraph Speed," BSTJ April 1924; "Certain Topics in Telegraph Transmission Theory," A.I.E.E. Trans. v. 47, April 1928) and Hartley ("Transmission of Information," BSTJ July 1928). Shannon says of them, in his second sentence, "A basis for such a theory is contained in the important papers of Nyquist and Hartley on this subject." Later footnotes reach R. C. Tolman's Principles of Statistical Mechanics (1938), Wiener's Cybernetics, W. G. Tuller's "Theoretical Limitations on the Rate of Transmission of Information" (Proc. IRE, May 1949), H. Sullivan, and Shannon's own "Communication in the Presence of Noise." Not one of those is in canon/, and I am not inventing ids for them. Nyquist 1928 and Hartley 1928 are the two real ancestors and a reading that wants the lineage should be told they are missing rather than handed a substitute.

A note on the other Shannon files. shannon-chess-1950 is in the canon and declares descends_from: [] on the same basis: its seven-item reference list contains no self-citation to the 1948 paper. So the canon now holds two Shannon entries, two years apart, by the same man at the same laboratory, with no citation running between them in either direction. That is a fact about the papers and not an oversight in either file. thorp-shannon-roulette-1961 names this entry directly — "shannon-limit-1948 is proposed in canon/proposals.md and unwritten" — and that sentence is now stale in the direction of being answered; I have not edited that file, because a canon job writes one file, and the descent index that build.py renders will link them without help.

What it is

In July and October of 1948, in two installments of the Bell System Technical Journal, a 32-year-old mathematician at Bell Telephone Laboratories published a paper that did for communication what thermodynamics did for heat engines: it replaced a collection of engineering rules of thumb with a quantity, a unit, and a bound.

The problem Shannon set himself is stated in his second paragraph, and the statement is famous enough to be worth having exactly:

> The fundamental problem of communication is that of reproducing at one point > either exactly or approximately a message selected at another point. Frequently > the messages have meaning; that is they refer to or are correlated according > to some system with certain physical or conceptual entities. These semantic > aspects of communication are irrelevant to the engineering problem. The > significant aspect is that the actual message is one selected from a set of > possible messages.

That last italicised phrase is the whole move. A message is not a thing with content; it is a choice out of a set. If you know the set and the probabilities, you can say how much choosing was done, in a unit — "bits, a word suggested by J. W. Tukey," as the paper notes, in what is very close to the word's first appearance in print.

Entropy. For a source producing symbols with probabilities p₁ … p_n, Shannon defines H = −Σ p_i log p_i, and proves in Theorem 2 that this is the only form satisfying three natural requirements (continuity, monotonicity in n for equiprobable events, and consistency under decomposing a choice into successive choices). He then does something almost nobody does with their own axiomatic characterisation. He disowns it:

> This theorem, and the assumptions required for its proof, are in no way > necessary for the present theory. It is given chiefly to lend a certain > plausibility to some of our later definitions. The real justification of these > definitions, however, will reside in their implications.

He also flags the name collision himself, in footnote 8, pointing at Tolman's statistical mechanics text: H "will be recognized as that of entropy as defined in certain formulations of statistical mechanics." Whether he should have used the word is section 4's problem, and it has been a problem for seventy-eight years.

Equivocation and capacity. Part II turns to the noisy channel. If the received symbol does not determine the transmitted one, the received bits are not all information; Shannon defines the equivocation H_y(x) — the remaining uncertainty about what was sent, given what was received — and the rate of actual transmission as R = H(x) − H_y(x). His worked example is memorable and cheap: transmit 1000 bits per second over a channel that flips 1 in 100 symbols, and the naive answer, 990, is wrong. It is wrong because it is not robust to the extreme case. Push the noise to the point where the received symbol is independent of the transmitted one and the naive rule credits you with 500 bits per second while "equally 'good' transmission would be obtained by dispensing with the channel entirely and flipping a coin at the receiving point." The equivocation correction gives 919 in the first case and 0 in the second, which is the behaviour a measure has to have. The capacity of the channel is then the maximum of R over all input sources: C = Max(H(x) − H_y(x)).

The theorem. Section 13 is titled "The Fundamental Theorem for a Discrete Channel with Noise," and it opens by naming the thing everyone expects to be true and that is not:

> One would expect, however, that to make this probability of errors approach > zero, the redundancy of the encoding must increase indefinitely, and the rate > of transmission therefore approach zero. This is by no means true.

Then, verbatim:

> Theorem 11: Let a discrete channel have the capacity C and a discrete > source the entropy per second H. If H ≤ C there exists a coding system such > that the output of the source can be transmitted over the channel with an > arbitrarily small frequency of errors (or an arbitrarily small equivocation). > If H > C it is possible to encode the source so that the equivocation is less > than H − C + ε where ε is arbitrarily small. There is no method of encoding > which gives an equivocation less than H − C.

Three sentences: achievability, a graceful-degradation clause, and the converse. Of the excess above capacity Shannon writes, in a line that has never been improved on, "Nature takes payment by requiring just that much uncertainty, so that we are not actually getting any more than C through correctly."

How he proved it, which is the interesting part. He did not build a code. He proved that one must exist by averaging over a whole population of them:

> The method of proving the first part of this theorem is not by exhibiting a > coding method having the desired properties, but by showing that such a code > must exist in a certain group of codes. In fact we will average the frequency > of errors over this group and show that this average can be made less than ε. > If the average of a set of numbers is less than ε there must exist at least one > in the set which is less than ε.

This is the random-coding argument, and it is the technique that made the theorem possible and the engineering impossible for forty-five years. Shannon knew it, and said so, in section 14 — a passage that matters enough to this file's section 3 that it is quoted there in full rather than here.

The Gaussian channel and the number people mean. Part IV specialises to the case every engineer actually has: a channel of bandwidth W perturbed by white thermal noise of power N, with the transmitter limited to average power P.

> Theorem 17: The capacity of a channel of band W perturbed by white thermal > noise power N when the average transmitter power is limited to P is given by > > C = W log (P+N)/N.

Shannon immediately restates it in operational terms: "This means that by sufficiently involved encoding systems we can transmit binary digits at the rate W log₂ (P+N)/N bits per second, with arbitrarily small frequency of errors. It is not possible to transmit at a higher rate by any encoding system without a definite positive frequency of errors." Written as C = W log₂(1 + P/N) this is the formula usually called the Shannon–Hartley theorem, Hartley's name attaching for the 1928 idea that capacity should go as the log of the number of distinguishable levels. Theorem 18 extends it to non-white noise with bounds in terms of entropy power, and Shannon is explicit that for arbitrary noise the maximisation "does not appear that [it] can be solved explicitly" — the general case is bounded, not solved.

The −1.59 dB. The figure most often quoted as "the Shannon limit" in coding practice is not in the 1948 paper. It falls out of Theorem 17 in the power-limited corner. Write η = R/W for spectral efficiency in bits per second per hertz and E_b/N₀ for energy per information bit over noise spectral density; then reliable transmission requires E_b/N₀ > (2^η − 1)/η, which decreases monotonically as η → 0 and approaches ln 2. In decibels, 10 log₁₀(ln 2) = −1.59 dB. Below that signal-to-noise ratio per bit, no bandwidth, no code, and no amount of computation gets a message across reliably. The presentation above is Costello and Forney's, who call it "the ultimate Shannon limit"; the content is Theorem 17's.

Rate–distortion, in passing. Part V asks what happens if you do not need exact reproduction — if a fidelity criterion is enough, as it is for speech, or images, or anything a human consumes. Theorem 21 gives the same shape of answer: a source has a rate R₁ for a given fidelity valuation, and it can be sent over a channel of capacity C at that fidelity if and only if R₁ ≤ C. This is the seed of rate–distortion theory and of every lossy codec since, and it matters here because it is the part of the paper that most resembles what a learned model does — throw away detail under a distortion budget — and therefore the part most often gestured at by people arguing about what models "really" do.

One thing the paper contains that its reputation does not. Section 17 gives a worked single-error-correcting code: seven bits, four of them message, three computed as parity checks, with the syndrome reading out the index of the failed bit. Shannon credits it in parentheses — "found by a method due to R. Hamming" — two years before Hamming published it himself. The paper that proved good codes must exist also printed the first good code, and the distance between those two facts is the history in section 3.

Why a reading would cite it

Three occasions, in descending order of how often they come up. All three are described as shapes, with dated instances named so a reading can find them; none of the instances is graded here, and nothing in this file is evidence.

1. When a claimed ceiling on AI needs to be measured against a real one. This is the main occasion and the reason the proposal called this "the model of an honest one." Claims of the form "there is a hard limit on what these systems can do, and here is the mathematics" arrive constantly, and a reading has to decide in a paragraph whether one is a result or a mood. Shannon supplies the checklist, because his limit has every property such a claim is implicitly borrowing:

• A stated model of the thing being limited. Shannon says exactly what a channel is: an input alphabet, an output alphabet, and transition probabilities, or in the continuous case a bandwidth and a noise spectrum. Change the model and the number changes, and he says so. Most AI ceiling claims never state the model at all; the object being bounded is "a language model," which is not a mathematical object.
• A converse with a proof. Not "we haven't seen it work," but "there is no method of encoding which gives an equivocation less than H − C," established from the definition of C.
• A matching achievability result, so the bound is tight — the limit is not merely an upper bound nobody knows how to reach but the exact edge of the possible.
• A measurable gap. Because C is a number in the same units as the achieved rate, at any moment in the last seventy-eight years you could say precisely how far the best working system was from the bound, in decibels, and watch that distance shrink. Section 3 gives the actual series: about 2.5 dB in the 1970s, 0.7 dB in 1993, 0.0045 dB in 2001.

Set that against the live instances. The "data wall" — the projection that the stock of public human-generated text will be exhausted by training runs somewhere between 2026 and 2032 (Villalobos et al., arXiv:2211.04325, Epoch AI) — is a careful, honest, and entirely different kind of claim: an extrapolation of two growth curves toward an intersection, with no converse, no proof, and explicit discussion of the things that would falsify it (synthetic data, transfer, data efficiency). Its authors do not call it a limit. Downstream summarisers do, and that is the slippage worth catching. Similarly for "On the Fundamental Limits of LLMs at Scale" (arXiv:2511.12869), named in perceptrons-1969 as an instance of that file's shape and named here for the same reason; and for the whole genre of "scaling has hit a wall" pieces. A reading that wants to say one of these is or is not an honest limit now has a specimen to hold it against, with the properties enumerated.

2. When infrastructure is the story. This is the least glamorous occasion and the most literal. Under the compute and infrastructure lens, frontier training is increasingly interconnect-bound rather than FLOP-bound, and the interconnect is a set of channels being engineered directly against this theorem. Coherent wavelength-division systems are reported to run very close to the fiber channel's Shannon limit, with 800G coherent optics shipping and 1.6T expected to follow. When a networking claim of that shape crosses a reading, this entry is the thing that lets it be read properly, including the caveat that matters: the linear AWGN capacity is what "close to the Shannon limit" refers to, and the capacity of an optical fiber as an actual physical channel is not known, because the Kerr nonlinearity makes it a nonlinear channel whose capacity remains open. The "nonlinear Shannon limit" quoted in the optics industry is an estimate under a nonlinear-interference model, not a theorem in Shannon's sense. That is a precision a reading can only get from having the original straight — and it cuts both ways, since "we exceeded the nonlinear Shannon limit" papers exist and are not the paradox they sound like.

3. When something is being measured in bits and the measurement is Shannon's. The training objective of every language model in production is cross-entropy, which is Shannon's H with a model's predicted distribution in place of the true one, reported in nats or bits per token; perplexity is its exponential. This is not an analogy. It is the same quantity, and the lineage runs through Shannon's own 1951 follow-up, "Prediction and Entropy of Printed English," in which human subjects guessing the next character put English somewhere between 0.6 and 1.3 bits per character — the first perplexity benchmark, run on people, seventy-five years ago. The irreducible loss term in scaling-laws-2020 — the constant that the power law approaches as parameters and data go to infinity — is interpreted in Kaplan et al. as the entropy of natural text. So when a reading covers a scaling result, or an argument about whether loss curves are bending, the units on the y-axis are this file's units, and the floor under the curve is claimed to be a Shannon entropy. Whether that floor is really an entropy of a well-defined source, as opposed to a fitted constant with a suggestive name, is an open and worthwhile question, and the honest version of it can only be asked by someone who knows what Shannon required of a source before H was defined for it: stationarity, ergodicity, and a probability model.

There is a fourth occasion I considered and rejected. Under the quantum lens the natural analogue is the Holevo bound, which limits classical information extractable from quantum states, and quantum channel capacities generally. Those are real, they descend from this, and they are not in this canon. Naming them here as though this entry covered them would be a citation a reading could not follow, so they are named as absent instead.

What it got right, and what it got wrong

Section 3 is not mandatory for limit, but this paper carries a dated engineering forecast about its own consequences, and a canon that only grades entries filed under prediction will systematically miss the forecasts that arrive embedded in engineering papers — the argument shannon-chess-1950 makes and this file adopts.

The theorem: standing, undefeated, 78 years

Claim (July 1948): reliable communication is possible at every rate below C and impossible at every rate above it.

Due: immediately and forever, as theorems are.

What happened: nothing. The converse has never been beaten and could not be — it follows from the definition of capacity — and the achievability half has been confirmed in the strongest possible way, by construction, repeatedly. What is worth recording is that the theorem constrained an industry for three quarters of a century and the industry never once found daylight in it. This is the base rate that matters when someone waves at a claimed limit on AI: the reason Shannon's holds is not that limits generally hold, it is that this one has a proof.

Shannon's own pessimism about codes: right for 45 years, then wrong, and right about why

This is the gradeable forecast, and it is in section 14 of the paper, immediately after the proof. In full:

> The demonstration of Theorem 11, while not a pure existence proof, has some of > the deficiencies of such proofs. An attempt to obtain a good approximation to > ideal coding by following the method of the proof is generally impractical. In > fact, apart from some rather trivial cases and certain limiting situations, no > explicit description of a series of approximation to the ideal has been found. > Probably this is no accident but is related to the difficulty of giving an > explicit construction for a good approximation to a random sequence.

Claim made: July 1948. Due by: open-ended — the claim is that the construction problem is hard for a structural reason, so it comes due whenever someone solves it.

What actually happened, as a series, from Costello and Forney's survey and the primary coding literature:

| year | scheme | gap to capacity | |---|---|---| | 1948 | the (7,4) Hamming code, in the paper itself | matched capacity only on a contrived channel | | 1968 | Pioneer: rate-1/2 convolutional, sequential decoding | ~3.3 dB real coding gain | | 1969 | Mariner: (32,6,16) biorthogonal code | ~2.2 dB real coding gain | | 1970s | NASA standard: rate-1/2 64-state convolutional inner + (255,223,33) Reed–Solomon outer | ~2.5 dB from capacity | | 1992 | Galileo, after the antenna failed: rate-1/6, 2¹⁴-state inner, Big Viterbi Decoder, variable-rate RS outer | 10.2 dB gain; P_b ≈ 2×10⁻⁷ at E_b/N₀ ≈ 0.8 dB | | 1993 | turbo codes (Berrou, Glavieux, Thitimajshima, ICC'93 Geneva) | 0.7 dB at P_b ≈ 10⁻⁵ (0.5 dB from the binary-input limit) | | 2001 | irregular LDPC (Chung, Forney, Richardson, Urbanke) | 0.0045 dB threshold; 0.040 dB simulated at block length 10⁷ | | 2009 | polar codes (Arıkan) | first explicit construction provably achieving capacity with low encoding and decoding complexity | | 2016 | 3GPP RAN1 #87, Reno: LDPC for 5G NR data channels, polar for control | the limit becomes a product specification |

Verdict: wrong, and it took forty-five years to become wrong. Shannon predicted that no explicit series of approximations to the ideal would be found; one was, in May 1993, and the community's first reaction was that it could not be real. The response widely reported from that period — "It can't be true; they must have made a 3 dB error" — comes to this file from a search summary rather than from a source I read, and is repeated here as folklore with that label on it.

But the second half of the sentence is the part to keep. Shannon guessed the obstacle was "the difficulty of giving an explicit construction for a good approximation to a random sequence," and that diagnosis was correct, including about the solution. Turbo codes work by making a convolutional code random-like — two recursive encoders separated by a large pseudorandom interleaver — while retaining enough structure to be iteratively decodable. LDPC codes are literally sparse random graphs with just enough regularity to admit belief propagation. The answer to "how do you explicitly construct something that behaves like a random code" turned out to be "build pseudo-randomness with a decodable skeleton," which is the thing Shannon said was hard, solved rather than circumvented. He named the right obstacle and underestimated the field's ability to get around it, which is a much better failure than naming the wrong obstacle.

One aggravating detail. Robert Gallager described low-density parity-check codes and an iterative decoder in his 1960 MIT thesis and 1962 paper. The work was, in Forney's phrase, "a bit of 21st-century coding that happened to fall in the 20th century," and it was largely forgotten for more than thirty years — rediscovered around 1996 by MacKay, and independently reached from expander graphs by Spielman. So part of Shannon's forty-five-year gap was not a gap in knowledge at all. The construction existed from 1962 and the field mislaid it. That is a fact about how research communities forget, and it is available to a reading that wants to say something about whether a current line of work is genuinely new.

What the paper claimed about its own scope: right, and ignored

Claim (July 1948, second paragraph): "These semantic aspects of communication are irrelevant to the engineering problem."

This is a scoping decision, not a metaphysical claim, and read as a scoping decision it was exactly right: excluding meaning is what made the theory possible, and no part of the capacity result needs a notion of meaning to be stated or proved. Graded as an engineering choice: correct, decisively, and it is hard to name a more productive act of deliberate narrowing in twentieth-century science.

Graded as a piece of communication, however, it failed, and Shannon watched it fail. Eight years later he wrote a one-page editorial in the IRE Transactions on Information Theory (vol. 2, no. 1, March 1956) titled "The Bandwagon":

> Information theory has, in the last few years, become something of a scientific > bandwagon. […] As a consequence, it has perhaps been ballooned to an importance > beyond its actual accomplishments. […] Applications are being made to biology, > psychology, linguistics, fundamental physics, economics, the theory of > organization, and many others. […] it is certainly no panacea for the > communication engineer or, a fortiori, for anyone else. […] the use of a few > exciting words like information, entropy, redundancy, do not solve all > our problems. […] Seldom do more than a few of nature's secrets give way at one > time.

This is the most quotable thing in this file and a reading should know it exists, because the pattern it describes — a technical result acquiring metaphorical reach far past its hypotheses, driven partly by the glamour of its vocabulary — is the pattern of section 4, and Shannon called it in 1956 against his own theory. He was right that it would happen; he was wrong, or at least unsuccessful, in thinking an editorial would stop it. Seventy years on, "entropy," "information," "signal," "bandwidth" and "compression" are still doing rhetorical work in AI argument that Shannon's definitions do not license.

The prediction he made about his own Theorem 2, which came true

A small one, and pleasing. Shannon said of his axiomatic derivation of H that it was "in no way necessary for the present theory" and given "chiefly to lend a certain plausibility." He was flagging, in advance, the result of his own that he expected to be over-weighted. It is the one that made it into the widest circulation and the one most often waved at by people who have not read past it. Authors are rarely that accurate about which of their results will be misused.

Commonly misused as

Required for the kind, and the section the entry exists for. Four misuses, then the one it has mostly escaped, then the template.

1. "Information-theoretic limits mean a model cannot produce anything that isn't in its training data"

The most common one in AI argument, and the most interesting, because unlike the Gödel misuse it is usually built out of something Shannon really wrote. The argument runs: information cannot be created by processing; a model is a processing of its training data; therefore a model's outputs contain no information beyond the training set; therefore nothing genuinely new can come out. Sometimes the data processing inequality is named explicitly. Sometimes it is just "information-theoretic limits."

The premise is real. Shannon states it, in Appendix 7:

> This means that any operation applied to the received signal, even though it > involves statistical elements, does not increase R.

That is the data processing inequality. In the modern form: if X → Y → Z is a Markov chain, then I(X;Z) ≤ I(X;Y). Post-processing cannot increase mutual information.

The conclusion does not follow, for three separate reasons, and they are worth keeping distinct.

First, the referent. The quantity that cannot increase is mutual information with respect to a specified source variable. Shannon's sentence is about information concerning the transmitted message, on a channel whose input the receiver is trying to recover. The inequality says: nothing you do at the receiver tells you more about what was sent. It says nothing whatsoever about whether the receiver's output is novel, useful, surprising, or correct with respect to anything other than the input — and "is this theorem true," "does this program compile," "is this protein stable" are not questions about the training corpus. A pocket calculator computing a product not present in its ROM violates no inequality; neither does a chess engine finding a move nobody has played; neither, whatever else is true of it, does a model emitting a sentence that is not in its training set.

Second, the direction of the constraint. The inequality bounds information about a source; it does not bound the entropy, the complexity, or the utility of the output. Deterministic post-processing of a fixed input produces something with less mutual information about that input than the input has about itself, and this is compatible with the output being longer, more surprising to a reader, or more valuable than the input. Compare godel-incompleteness-1931, where the misuse also works by silently swapping the object the theorem quantifies over. The move is the same move.

Third, computation is not in the theory at all. Shannon's channel model has no notion of a function being computed, a program being run, or an inference being drawn. Every quantity in the paper is defined over probability distributions on symbol sequences. Whether a fitted function can compute something its training data does not literally contain is a question in learning theory and computability — generalisation, extrapolation, inductive bias — and information theory is simply silent on it. Importing a channel theorem into that question is a category error, and it is a category error in the same family as the one section 4 of turing-halting-1936 documents.

Where a version of the argument does have force, and a reading should not overcorrect: information-theoretic bounds are legitimately used to argue that a model with finite capacity cannot represent every distinction a task requires, or that a training signal carrying k bits cannot specify a behaviour needing more than k bits to describe. Those are real arguments with stated models. The tell that separates them from the misuse is whether the arguer has said what the source variable is. If they have not, the inequality they are invoking has no referent and is doing rhetorical work only.

2. "Compression is understanding" — and its mirror, "entropy measures meaning"

The claim that the ability to compress is the same thing as the ability to understand has a serious version and a sloppy one, and Shannon licenses neither.

The serious version — prediction and compression are formally interchangeable, so a better language model is a better compressor and vice versa — is true, and it is true because of Shannon's source coding results, not his channel results. The Hutter Prize (2006–) is built on it. The step that Shannon does not license is the next one: that compressing well is understanding, or that compression ratio measures intelligence. Shannon's H is defined for a source with a probability model; it is invariant to everything about the message except its distribution over a set. The standard demonstration is that a meaningful sentence and a scrambled permutation of its characters have the same character-level Shannon entropy — a fact about the measure that is fatal to any reading of it as a measure of order, structure, or sense. Shannon's entropy does not look inside the message. Algorithmic complexity (Kolmogorov, Chaitin) does look inside, and is a different quantity with different theorems; the two are routinely conflated in exactly the arguments where the conflation matters most.

The mirror error runs the other way: treating a low-entropy or highly compressible output as evidence of vacuity, or a high-entropy one as evidence of creativity. Both directions import a theorem about codes into a claim about cognition.

The naming is a large part of why this happens, and the story is worth carrying with the caveat. Myron Tribus reported, from a 1961 conversation with Shannon, that Shannon considered calling the quantity "information," rejected it as overused, considered "uncertainty," and was told by von Neumann: "You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage!" The anecdote is second-hand, published in Tribus and McIrvine's Scientific American piece of September 1971, and rests on one man's recollection of a conversation with a third party. It should be told as a story, not cited as a fact — but if it is even approximately true, the founder of the field was warned at the naming ceremony that the word would be used to win arguments, and took the word anyway.

3. "Shannon proved that meaning is irrelevant to information"

The strong form: information theory shows semantics is not real, or not needed, so a system that manipulates symbols statistically is doing everything there is to do. The opposite strong form, deployed by the other side of the same argument: Shannon proved that information has nothing to do with meaning, therefore statistical language models can never mean anything.

Both misread a scoping sentence as a metaphysical one. "These semantic aspects of communication are irrelevant to the engineering problem" defines what Shannon will and will not address. It is not a finding, it has no proof, and it is not even a claim about the world — it is a claim about which problem the paper is solving. Read it as its author wrote it and both arguments evaporate: neither side gets to recruit him.

Some of the responsibility here is not Shannon's. The 1949 book that put the theory in front of a general readership, The Mathematical Theory of Communication (note the definite article, replacing his indefinite one), paired Shannon's paper with an expository essay by Warren Weaver that proposed three levels of communication problem — technical, semantic, and effectiveness — and speculated about the theory reaching the second and third. That framing did much of the work of licensing seventy years of extension into linguistics, biology, and psychology, and it is Weaver's, not Shannon's. I have not read Weaver's essay and this paragraph is built from its widely reported structure plus the change in title, which I verified; a reading that wants to assign responsibility should read it first.

4. "The Shannon limit" as a rhetorical stand-in for any ceiling

The cheapest and most common misuse, and the hardest to argue with because it is not really an argument: "we're approaching the Shannon limit of X," where X is transistor density, human attention, model scale, organisational bandwidth, or the economy. The phrase is being used to mean "a hard physical ceiling that cleverness cannot beat," and to borrow the authority of a theorem for a claim that has none of the theorem's structure.

The test is mechanical and a reading can apply it in one line: name the channel, name the noise, and state the number in its units. Shannon's limit is not a vibe about diminishing returns; it is C, in bits per second, computable from a bandwidth and a signal-to-noise ratio, against which any candidate system's achieved rate can be placed. If the person invoking it cannot produce those three things, they have borrowed a name and left the content behind. Note that this test is generous — it does not require a proof, only a model and a number — and almost nothing in AI discourse that invokes "information-theoretic limits" passes even that.

The misuse it has largely escaped, and why that is instructive

Almost nobody has stretched the channel coding theorem into a claim about minds. There is no Shannon equivalent of Lucas and Penrose on Gödel, no cottage industry arguing that channel capacity proves machines cannot think. The proposal for this entry noticed this and it is worth explaining rather than merely observing, because the explanation is a diagnostic a reading can reuse.

Gödel's theorems are about formal systems, and "the mind" can be presented, with some plausibility, as a formal system; that presentation is where the misuse gets its foothold, and refuting it requires arguing about what the mind is. Shannon's theorem is about a channel — an input alphabet, an output alphabet, transition probabilities — and no one has managed to make "the mind" look like a channel in a way that survives contact with the definition. The hypotheses are too concrete to be smuggled past. The general lesson: a limit result gets misused in proportion to how abstract its hypotheses are. Concrete hypotheses are misuse-resistant. When a reading meets a claimed limit stated over an object as vague as "a sufficiently large model" or "any system trained on human data," the vagueness is not incidental to its rhetorical appeal — it is the mechanism.

There is one genuine scope caveat that the theorem's admirers do under-report, and it belongs here rather than in the misuse list because it is a limitation rather than a distortion. Capacity is asymptotic. Theorem 11's guarantee is a statement about codes of increasing block length; the coding-theoretic headlines in the table above are thresholds at block lengths of 10⁵ to 10⁷, with latency to match. At short block lengths the achievable rate falls measurably below C, by an amount characterised in the finite-blocklength literature (Polyanskiy, Poor and Verdú, 2010, which I have not read) and which matters enormously in low-latency regimes. "Within 0.0045 dB of capacity" is true and is a statement about very long codes. Anyone quoting it about a real-time system should say so.

The template, which is why the entry is worth keeping

Strip out the specifics and what remains is a five-part shape a reading can apply to any claimed limit in a paragraph:

1. What is the object? Shannon: a channel with a stated alphabet and transition probabilities. If the object is "an AI system," there is no theorem yet, only a topic. 2. What is the quantity, and in what units? Shannon: bits per second, or bits per channel use. A limit whose quantity cannot be measured cannot be approached, and a limit that cannot be approached cannot be tested. 3. Is there a converse, and is it proved? Shannon: yes, from the definition. Most claimed limits have an intuition here and call it a proof. 4. Is there a matching achievability result? Shannon: yes, which is what makes the bound tight rather than merely an upper bound. This is the property almost no borrowed limit has, and its absence is what turns "we don't know how" into "it can't be done." 5. What is outside the hypotheses? Shannon: computation, meaning, inference, learning, and the value of what is transmitted. Every misuse in this section works by moving one of those five things inside.

Point 4 is the one to press hardest, because it is where the two-sidedness pays off. Shannon's theorem was, for its first forty-five years, mostly cited for the half that turned out to be an understatement — the promise, not the prohibition. A limit that also tells you what is achievable is a research programme; a limit that only forbids is a stopping point. When a reading meets a claimed AI limit that only forbids, the honest observation is not that it is wrong. It is that it is a different and lesser kind of object than the one whose name it has borrowed.

Sources

Primary, read directly.

• C. E. Shannon, "A Mathematical Theory of Communication," Bell System Technical Journal, vol. 27, pp. 379–423 and 623–656, July and October 1948. Read as the 55-page corrected reprint hosted at people.math.harvard.edu, which is the standard scan of the corrected version and carries the line "Reprinted with corrections from The Bell System Technical Journal" on page 1. Pages read in full: 1–4 (Introduction, Part I §1–2), 10–11 (§6, Theorem 2 and its disclaimer, the Tolman footnote), 19–28 (Part II in its entirety: §11–17, Theorems 10, 11, 12, the Discussion of §14, the Hamming code of §17), 39–45 (Theorems 14–19, Part IV, including Theorem 17 and the "loc. cit." footnote to "Communication in the Presence of Noise"), 50–51 (Theorems 21, 22, 23), 54–55 (Appendix 7, including the data processing statement, and the end of the paper). Every verbatim quotation from the 1948 paper in this file was transcribed from those pages. Pages 5–9, 12–18, 29–38, 46–49 and 52–53 were not read: they contain the Markov models of English, the source coding theorems of Part I, the mathematical preliminaries of Part III, and Appendices 1–6, none of which this entry makes a claim about. The paper has no bibliography — all references are footnotes, and the footnote list in the opening section above is complete for the pages read.
• C. E. Shannon, "The Bandwagon," IRE Transactions on Information Theory, vol. 2, no. 1, March 1956, p. 3. Read via the full text at reimbar.org; the IEEE Xplore record (document 1056774) returned no body text. All five quoted fragments were confirmed against that text. It is a one-page editorial and the fragments are a substantial portion of it; a reading quoting it should read the page rather than this file.

Secondary, read directly.

• D. J. Costello, Jr. and G. D. Forney, Jr., "Channel Coding: The Road to Channel Capacity," submitted to Proceedings of the IEEE, first revision November 2006, arXiv:cs/0611112. Pages 1–3, 14–19 and 22–25 read directly. Source for: the ln 2 (−1.59 dB) ultimate limit and its derivation from η; the McEliece Encyclopedia Galactica passage from his 2004 Shannon Lecture; the deep-space series (Mariner 1969, Pioneer 1968, the 1970s NASA standard concatenated code and its ~2.5 dB gap, Galileo circa 1992); the turbo code figures (0.7 dB from the η = 1 Shannon limit, 0.5 dB from the binary-input limit, at P_b ≈ 10⁻⁵, interleaver N = 2¹⁶); the Gallager LDPC dormancy and the "21st-century coding that happened to fall in the 20th century" characterisation; the Chung et al. figures (0.0045 dB threshold, 0.040 dB simulated at block length 10⁷). This is a survey by two of the field's principals, one of them a co-author of the 0.0045 dB result, and it is the best single secondary source for the whole arc.

Identified from search results and not read. Named as citations, not summarised as findings; a reading that wants any of them should read it first.

• C. Berrou, A. Glavieux and P. Thitimajshima, "Near Shannon Limit Error-Correcting Coding and Decoding: Turbo-Codes (1)," Proc. ICC'93, Geneva, 23–26 May 1993, pp. 1064–1070. The 0.7 dB figure is taken from Costello and Forney, not from this paper; a direct fetch of a hosted copy returned HTTP 403.
• S.-Y. Chung, G. D. Forney, T. J. Richardson and R. Urbanke, "On the Design of Low-Density Parity-Check Codes within 0.0045 dB of the Shannon Limit," IEEE Communications Letters, vol. 5, no. 2, pp. 58–60, 2001.
• E. Arıkan, "Channel Polarization: A Method for Constructing Capacity-Achieving Codes for Symmetric Binary-Input Memoryless Channels," IEEE Trans. Inf. Theory, vol. 55, no. 7, July 2009, pp. 3051–3073.
• R. G. Gallager, "Low-Density Parity-Check Codes," 1962 (and the 1963 MIT Press monograph); D. J. C. MacKay and R. M. Neal's mid-1990s rediscovery.
• Y. Polyanskiy, H. V. Poor and S. Verdú, "Channel Coding Rate in the Finite Blocklength Regime," 2010 — the source for the asymptotic caveat in section 4.
• C. E. Shannon, "Prediction and Entropy of Printed English," BSTJ vol. 30, pp. 50–64, 1951. The 0.6–1.3 bits per character range is from a search summary.
• W. Weaver's expository essay in Shannon and Weaver, The Mathematical Theory of Communication (University of Illinois Press, 1949). Not read; see the flag in section 4.3.
• R.-J. Essiambre et al. on the capacity limits of optical fiber networks, for the nonlinear-capacity caveat in section 2. Abstract-level only.

Search-result summaries, relied on for dates and figures only.

• The 3GPP RAN1 #87 decision (Reno, November 2016): LDPC for 5G NR eMBB data channels, polar codes for control channels.
• P. Villalobos et al., "Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data," arXiv:2211.04325 (Epoch AI) — the 2026–2032 range for the stock of public human text. Named in section 2 as an instance of a shape, explicitly not graded here.
• M. Tribus and E. C. McIrvine, "Energy and Information," Scientific American, September 1971 — the von Neumann naming anecdote, carried with its second-handedness stated in the text.
• Reports of coherent WDM systems operating close to the fiber Shannon limit, and of 800G coherent optics shipping as of late 2025, used in section 2 with the nonlinear caveat attached.
• The "It can't be true; they must have made a 3 dB error" reaction to the 1993 turbo code result, attributed in section 3 to folklore rather than to a source.

Encountered and deliberately not used. A large volume of recent arXiv material applies information-theoretic language to language models — entropy dynamics during training, "semantic information theory for LLMs," epiplexity, information-theoretic reliability ceilings. Some of it is careful. None of it was read, and none of it is characterised in this file, because a limit entry's job is to state what the 1948 theorem says and what it does not, and a survey of 2025–2026 papers that use its vocabulary would be a reading, not a canon entry.

On the boundary. This file cites the canon's own godel-incompleteness-1931, turing-halting-1936, perceptrons-1969, shannon-chess-1950, thorp-shannon-roulette-1961 and scaling-laws-2020 as related entries, and names Nyquist 1928 and Hartley 1928 as real ancestors that are absent from canon/ rather than inventing ids for them. It deposits nothing in the evidence ledger, grades no model, places no needle and no landmark. It grades exactly one dated claim — Shannon's own 1948 remark about explicit code construction — and that claim is his, about coding theory, and forty-five years settled. Its assertions about the present state of AI are limited to naming two papers and one class of infrastructure claim as instances of shapes, evaluated in neither direction.