Information Theory

Syllabus: Information Theory: Entropy, mutual information and channel capacity theorem.

5 hand-picked visual resources on this topic, 3 of them little-known gems — under each card, look for See it. Every link is checked against the live site before it ships.

Entropy and mutual information

Entropy is the average bits per symbol a source truly needs. It is largest when all symbols are equally likely and zero when the source is deterministic — so any question asking to maximise $H$ is answered by equiprobability.

$H(X)=-\sum_i p_i\log_2p_i\ \ \text{bits/symbol}$
$0\le H\le\log_2M$
$R=rH(X)\ \ \text{bits/s}$
$r$ = symbol rate
$I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$
$H(X,Y)=H(X)+H(Y|X)$
chain rule
$\eta=\dfrac{H}{\bar L},\qquad H\le\bar L<H+1$

Where marks are lost

Use $\log_2$ . A natural log gives nats and an answer off by $\ln2=0.693$ — a very common and very avoidable error.

See it

Channel capacity

Shannon's theorem sets the ceiling: reliable transmission is possible at any rate below $C$ and impossible above it. Bandwidth and SNR trade against each other, but not equally — capacity grows only logarithmically with power.

$C=B\log_2\!\left(1+\dfrac{S}{N}\right)\ \text{bits/s},\qquad N=N_0B$

← Analog Modulation Digital Communication →

Loading revision…