Entropy and Channel Capacity
Information is surprise. A noisy channel does not condemn you to errors — it condemns you to a lower rate, which is a far stranger claim than it sounds.
Skip to the animationInformation is surprise, entropy is average surprise, and channel capacity C = B·log₂(1 + S/N) is the rate above which no code can communicate reliably — while below it, an arbitrarily low error rate is achievable, which is a far stronger claim than it first appears.
Information as surprise
Shannon defined information as I = log₂(1/p) — the less likely a message, the more it tells you. A certain event carries zero information. The definition deliberately ignores meaning, which is what made a mathematical theory possible.
Entropy H = −Σ p·log₂p is the average information per symbol. For a binary source it peaks at exactly 1 bit when the outcomes are equally likely, and falls to zero at either extreme. Bias means predictability, and predictability means less information.
The source coding theorem
No lossless code can average fewer bits per symbol than the entropy. English runs about 1.5 bits per letter against a naive 4.7, and that gap is exactly what a compressor exploits.
It also means a compressor cannot shrink *every* file — that would violate the theorem, and a simple counting argument confirms it. Huffman coding comes within one bit of the entropy, which is provably the best any prefix code can do.
Channel capacity
Shannon-Hartley: C = B·log₂(1 + S/N). Bandwidth enters linearly and signal-to-noise logarithmically, so doubling the spectrum doubles capacity while doubling the power adds only about one bit per hertz.
| Change | Effect on capacity |
|---|---|
| Double the bandwidth | Doubles |
| Double the power | Adds ≈ 1 bit/s/Hz |
| Hundredfold power increase | Under 7 bits/s/Hz |
| Halve the noise | Adds ≈ 1 bit/s/Hz |
That asymmetry explains almost every decision in modern radio: 5G moving to millimetre wave, Wi-Fi channels widening, and national spectrum auctions raising billions. Bandwidth is the scarce and valuable input.
The noisy channel coding theorem
Below capacity, an arbitrarily low error rate is achievable with sufficient coding. Above it, no code works at all. A noisy channel does not condemn you to errors — it condemns you to a lower rate.
Shannon proved such codes exist without constructing any. Finding practical ones took until the 1990s, with turbo codes and then LDPC.
Where real systems sit
Modern LDPC codes run within a fraction of a decibel of the bound, so coding is close to finished as a source of gain. The remaining levers are bandwidth, antenna gain and spatial reuse.
MIMO appears to beat the limit and does not — it creates several parallel channels, each subject to the same bound. Multiplying the number of channels is allowed; exceeding the capacity of one is not.
The numbers you will be asked for
- Information content
I = log₂(1/p) bits
- Entropy
H = −Σ pᵢ log₂ pᵢ
- Shannon-Hartley
C = B · log₂(1 + S/N)
- Capacity in dB terms
C ≈ B · SNR_dB / 3
a useful rule of thumb
- Shannon limit
Eb/N0 ≥ −1.59 dB
the absolute floor, at infinite bandwidth
- Code rate
R = k/n
k data bits in n transmitted bits
Watch it work
Check yourself
question 1 / 4
One question at a time. Pick an answer to see why it is right or wrong, then move on — there is no score to keep and nothing is saved.