Contents
Figure 1: Entropy, KL Divergence, and Mutual Information Explained for ML — a fair die is the cleanest picture of entropy: six equally likely outcomes, log₂6 ≈ 2.585 bits of uncertainty, and no way to compress it
You call CrossEntropyLoss every day, you have seen "KL divergence" in a VAE paper, and you have probably used "mutual information" in a feature selection tutorial. Do those three ideas connect? They do, and once you see how, a lot of machine learning stops feeling like a pile of unrelated formulas.
All three answer one question: how surprised should you feel? Entropy measures the surprise in the truth. Cross-entropy measures the surprise you get when you describe the truth with the wrong model. KL divergence is the difference. Mutual information is the surprise that disappears when you learn something about another variable.
The One Question Behind All Three
Start with a single event. If it has probability p, the surprise it carries is:
−log p
A certain event carries zero surprise. A rare event carries a lot. Every quantity in this article is an average of that one term, weighted by how often each outcome occurs.
| Quantity | What it averages | Formula |
|---|---|---|
| Entropy | surprise under the true distribution | H(P) = −Σ P log P |
| Cross-entropy | surprise under the model, weighted by the truth | H(P,Q) = −Σ P log Q |
| KL divergence | the excess surprise | KL(P‖Q) = Σ P log(P/Q) |
| Mutual information | surprise removed by knowing X |
`I(X;Y) = H(Y) − H(Y |
That is the whole article in one table. The rest is what each row means, when it breaks, and how to compute it without tripping over log bases.
Entropy: Measuring Uncertainty
Entropy is the average surprise of a distribution:
H(P) = − Σₓ P(x) · log P(x)
A rare event surprises you a lot, so it contributes a large amount of information; a common event barely registers. Entropy weights each surprise by how often it happens, so it is an expected value over the distribution, not a property of any single outcome.
The Twenty Questions framing is exact, not a metaphor. If outcomes are uniform over 2ᵏ possibilities, the optimal yes/no strategy needs exactly k questions, and the entropy is k bits. Entropy is the average number of yes/no answers you cannot avoid.
Three anchors worth memorising:
| Distribution | Entropy (bits) | Entropy (nats) |
|---|---|---|
Always heads (p = 1) |
0 | 0 |
Fair coin (p = 0.5) |
1 | 0.693 |
Biased coin (p = 0.9) |
0.469 | 0.325 |
A fair coin is the maximum for two outcomes. A coin that always lands heads carries zero bits: you never feel surprised. And a uniform distribution maximises entropy over a fixed finite set of outcomes — total uncertainty means you cannot predict anything, and no other distribution on the same support is harder to guess.
That qualifier matters. The maximum-entropy statement is about a fixed finite support. For continuous variables the analogue is differential entropy, which can be negative and depends on scale — it is not a count of bits.
Bits vs Nats
The log base is a unit choice, nothing more. Base 2 gives bits, base e gives nats, and:
1 nat = 1 / ln 2 ≈ 1.4427 bits
So a value in bits is about 44% larger than the same value in nats. That is a multiplicative conversion, not an additive difference: 0.693 nats = 1 bit, and 1 nat = 1.44 bits.
PyTorch and TensorFlow use natural logs, which is why a loss value from your framework will not match a textbook example computed in bits. Multiply nats by 1.4427 to compare with the textbook, or divide bits by the same number. For language models, perplexity is exp(cross-entropy) when the loss is in nats — another place the base silently matters.
Cross-Entropy: The Cost of Using the Wrong Model
Now suppose the world follows P, but your model believes Q:
H(P, Q) = − Σₓ P(x) · log Q(x)
The coding reading is the cleanest one. An optimal code for Q assigns the event x a length of −log Q(x). If the data actually comes from P, the expected code length is H(P,Q) — the average number of units you spend describing reality with a code built for a belief that is not reality.
If Q = P, cross-entropy equals entropy and you pay the minimum possible cost. If Q is wrong, you pay extra, and the extra is exactly the KL divergence.
Cross-entropy is not symmetric either. H(P,Q) ≠ H(Q,P) in general, for the same reason KL is not.
Why It's Your Classification Loss
For a classifier, the true label is one-hot. P puts all its mass on one class y, so the sum collapses to a single term:
H(P, Q) = − log q_y
That is the whole loss. Give the correct class probability 0.9 and you pay 0.105 nats; give it 0.01 and you pay 4.605 nats. The penalty is unbounded as the predicted probability goes to zero, which is why a confidently wrong prediction dominates the gradient.
Minimising cross-entropy is maximising the likelihood of the training data. For one-hot labels, −log q_y is the negative log-likelihood of the observed class, so the classifier has been doing maximum likelihood all along — I went through that equivalence in maximum likelihood estimation [2].
Binary cross-entropy and logistic loss are the same quantity written two ways. And CrossEntropyLoss in PyTorch expects logits, not probabilities — it applies log-softmax internally, which is numerically safer than computing softmax and then taking a log.
KL Divergence: The Extra Cost
Subtract the entropy from the cross-entropy and the excess is the KL divergence:
KL(P ‖ Q) = Σₓ P(x) · log [ P(x) / Q(x) ]
The identity that makes cross-entropy make sense is:
H(P, Q) = H(P) + KL(P ‖ Q)
One line of algebra proves it:
−Σ P log Q = −Σ P log P + Σ P log(P/Q)
Entropy is the unavoidable cost of real uncertainty; KL is the extra cost of using the wrong model. When the training labels are fixed, H(P) does not depend on the model, so minimising cross-entropy is exactly minimising KL(P‖Q) [2]. Every classifier trained with cross-entropy is secretly pushing its predicted distribution toward the empirical one.
Properties You Should Know
- Non-negative.
KL(P‖Q) ≥ 0always — Gibbs' inequality [1]. - Zero only when identical.
KL(P‖Q) = 0exactly whenP = Qalmost everywhere. - Asymmetric.
KL(P‖Q) ≠ KL(Q‖P)in general. - Not a distance. It fails symmetry and the triangle inequality, which is why it is called a divergence.
- Can be infinite. If
Q(x) = 0whereP(x) > 0, the log ratio blows up. - Not bounded above. Even between distributions on the same finite set, KL can be arbitrarily large.
If you need a symmetric, bounded alternative, Jensen-Shannon divergence is the standard one:
JSD(P ‖ Q) = ½ KL(P ‖ M) + ½ KL(Q ‖ M), with M = ½(P + Q)
JSD is bounded between 0 and log 2 in nats — between 0 and 1 in bits [1]. It is not itself a metric, but its square root is, which is the version to use if you need the triangle inequality.
Why the Direction Matters
The asymmetry is not a technicality. It changes what a fit does.
KL(P‖Q)penalisesQfor putting low probability wherePhas mass. To avoid the penalty,Qmust cover every mode ofP. This is mass-covering, zero-avoiding, and mean-seeking — the fit spreads out and blurs across modes.KL(Q‖P)penalisesQfor putting mass wherePhas little.Qcan safely ignore a mode ofPas long as it fits one of them well. This is zero-forcing and mode-seeking — the fit locks onto a single mode and looks too narrow.
Variational inference minimises the second form, KL(q‖p) with q the approximation and p the posterior [1]. That is exactly why VI approximations are famously overconfident and too narrow: the objective rewards picking a mode over covering the distribution. If you have seen the KL penalty in RLHF, the direction and estimator details are a separate story — KL divergence in RLHF covers that end.
A naming warning: "forward" and "reverse" KL are used inconsistently across fields. Read the arguments, not the label.
Where KL Shows Up in ML
- Variational autoencoders. The loss is a reconstruction term plus
KL(q(z|x) ‖ p(z)), which pulls the encoder's posterior toward theN(0, I)prior. That KL term is what makes the latent space smooth enough to sample from. - Variational inference. The ELBO is
log p(x) − KL(q(z)‖p(z|x)), so maximising the ELBO is minimising the KL to the posterior. Same identity, different clothes [2]. - Knowledge distillation. The student minimises
KL(p_teacher ‖ p_student)at a temperature, which is why soft targets carry more signal than hard labels. - Reinforcement learning. TRPO constrains the KL between the old and new policy; PPO approximates the same idea with a clipped ratio; RLHF adds an explicit KL penalty to a reference model.
- Regularisation. A KL term between the posterior and a prior is a soft constraint, and a Gaussian prior recovers L2 — the same connection as MLE versus MAP.
Sometimes you want KL small, sometimes you read it as a signal. In some settings a large KL means surprise, disagreement, or information gained — but the number is only meaningful next to the distributions it was computed from.
Mutual Information: What One Variable Tells You About Another
Mutual information asks how much knowing X reduces your uncertainty about Y:
I(X; Y) = H(Y) − H(Y | X) = H(X) + H(Y) − H(X, Y)
The conditional entropy H(Y|X) is the uncertainty left in Y after you observe X. If X tells you nothing, H(Y|X) = H(Y) and the mutual information is zero. If X determines Y perfectly, H(Y|X) = 0 and the mutual information equals H(Y).
The KL reading is the one that generalises:
I(X; Y) = KL( P(X, Y) ‖ P(X) P(Y) )
MI is the divergence between the true joint distribution and the product of the marginals. Independence is exactly the product distribution, so MI measures how far your variables sit from independent [2]. It inherits non-negativity from KL, and it is symmetric because the joint and the product of marginals are.
Useful consequences:
I(X;Y) ≥ 0, with equality exactly whenXandYare independent.- It is symmetric:
I(X;Y) = I(Y;X), so it cannot tell you direction or causation. - For discrete variables,
I(X;Y) ≤ min(H(X), H(Y))— shared information cannot exceed either variable's uncertainty. - The chain rule:
H(X,Y) = H(X) + H(Y|X).
One caveat that bites in practice: for continuous variables, entropy is differential entropy and can be negative, but mutual information is still non-negative. The "bits" interpretation does not carry over cleanly; the estimator does.
Why MI Beats Correlation for Feature Selection
Pearson correlation measures linear association only. MI measures any statistical dependence. A feature with a U-shaped relationship to the target can have correlation near zero and mutual information well above zero — the feature is informative, but its effect cancels in a straight-line fit.
That is the pitch for mutual_info_classif and mutual_info_regression. Rank features by MI with the target, keep the top ones, and you catch relationships a correlation matrix would drop.
But MI is not a free lunch:
- It is not adjusted for cardinality. A high-cardinality identifier-like feature can score high by memorising noise; MI has no built-in complexity penalty.
- The continuous estimator is a k-nearest-neighbour estimate, with
n_neighbors=3by default — higher values reduce variance and add bias [4]. discrete_features="auto"treats dense input as continuous. Integer-coded categorical columns will be fed to the continuous estimator unless you pass a boolean mask [4].- The values are not comparable across datasets or sample sizes. Treat them as rankings within one run, not as absolute information.
MI also cannot tell you direction. It is symmetric, so it can say "these are dependent" but never "X causes Y". For a broader comparison of selection strategies, see feature selection: filter, wrapper, embedded.
Entropy in Decision Trees
A decision tree split is an information-gain calculation. At each node it picks the feature that reduces label uncertainty the most:
IG(Y, X) = H(Y) − H(Y | X) = I(Y; X)
That is mutual information between the feature and the label, in disguise. The greedy split search evaluates candidate thresholds, computes the weighted child entropies, and keeps the split with the largest gain.
Two details the textbooks blur:
- ID3 and C4.5 use entropy; CART defaults to Gini impurity. scikit-learn's
DecisionTreeClassifierusescriterion="gini"by default — passcriterion="entropy"if you want the information-gain version [3]. Gini is cheaper (no logarithm) and usually behaves similarly, but it is a different impurity. - The search is greedy and local. A split that maximises immediate information gain can be globally suboptimal; that is why ensembles and pruning exist.
All Three in Code
import numpy as np
from scipy.stats import entropy
from sklearn.feature_selection import mutual_info_classif
p = np.array([0.7, 0.2, 0.1]) # "true" distribution
q = np.array([0.5, 0.3, 0.2]) # model's distribution
H_p = entropy(p) # nats by default
H_pq = -(p * np.log(q)).sum() # cross-entropy, nats
kl = entropy(p, q) # KL(P || Q), nats
print(H_p, H_pq, kl)
print(np.isclose(H_pq, H_p + kl)) # True
print(entropy(q, p)) # different: KL is asymmetric
# The same numbers in bits
print(entropy(p, base=2), entropy(p, q, base=2))
# Mutual information: nonlinear dependence that correlation misses
rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 2000)
y = (x**2 + 0.05 * rng.normal(size=2000) > 0.3).astype(int)
print(np.corrcoef(x, y)[0, 1]) # near zero
print(mutual_info_classif(x.reshape(-1, 1), y, random_state=0)) # clearly positive
Run it and check four things.
- The identity holds numerically:
H_pqequalsH_p + klto machine precision. - Swapping the arguments changes the answer, which is the asymmetry in one line.
- The base changes the units, not the ordering.
base=2multiplies every value by1.4427. - Correlation misses the U-shape while MI catches it.
Two API details worth knowing. scipy.stats.entropy normalises its inputs if they do not sum to 1 [5] — convenient, but it will silently hide a bug where you forgot to normalise. And mutual_info_classif adds small noise to continuous features to break ties, which is what random_state controls [4].
The Cheat Sheet
- Entropy: how uncertain is the truth?
H(P) = −Σ P log P - Cross-entropy: what does it cost to describe the truth with my model?
H(P,Q) = −Σ P log Q - KL divergence: how much extra do I pay for the wrong model?
KL(P‖Q) = H(P,Q) − H(P) - Mutual information: how much does
Xtell me aboutY?I(X;Y) = H(Y) − H(Y|X) - Conditional entropy: how much uncertainty remains in
Yafter I seeX?H(Y|X) = H(X,Y) − H(X)
Common Pitfalls
- Mixing log bases. A value in bits is
1.4427×the same value in nats — a 44% increase. Pick one base and stay in it. - Treating KL as a distance. It is asymmetric and violates the triangle inequality; the order of arguments is part of the meaning.
- Forgetting zero probabilities. A zero in
QwherePhas mass makes KL infinite. Smooth or clip, and check forinfbefore you log a metric. - Feeding logits into a probability formula. Apply softmax first, or use a loss that does it internally —
CrossEntropyLossexpects logits,NLLLossexpects log-probabilities. - Trusting MI estimates on small continuous samples. The k-NN estimator is noisy; treat the values as rankings, not measurements.
- Leaving
discrete_features="auto"on dense categorical data. Integer codes get the continuous estimator and the scores become meaningless. - Assuming MI shows direction. It is symmetric; it cannot tell you whether
XcausesY. - Calling
entropy(p, q)and assuming it normalises the way you meant. It normalises each input separately, which can mask a preprocessing bug. - Assuming scikit-learn's trees use entropy. The default is Gini; information gain needs
criterion="entropy".
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Information Theory, Inference, and Learning Algorithms | entropy and KL from first principles, then the inference and machine-learning chapters that use them. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read chapters 1 and 2 for the coding interpretation, then chapter 4 for the KL divergence and its role in inference. MacKay's derivations are unusually geometric, and the book is free on his site if you want to try before buying.
Wrapping Up
Entropy measures uncertainty. Cross-entropy measures the cost of describing reality with an imperfect model. KL divergence is the gap between them, and that gap is why cross-entropy loss works. Mutual information measures shared information and gives you a dependence measure that correlation cannot provide.
The identity to remember is H(P,Q) = H(P) + KL(P‖Q). Everything else follows: fixed labels make the entropy term constant, so training on cross-entropy is training on KL; and mutual information is the same divergence applied to a joint distribution versus its marginals.
The experiment worth running: take a classifier you trained recently, compute its average cross-entropy, the entropy of its label distribution, and the difference. That difference is the model's KL divergence from the empirical truth, and watching it shrink as training proceeds makes the theory concrete.
If the probability foundations still feel shaky, math for machine learning maps what actually matters, and MLE versus MAP shows how the same log-probability reasoning turns into a regulariser.
Now go measure something.
Frequently Asked Questions
Is cross-entropy the same as KL divergence?
No. Cross-entropy equals entropy plus KL divergence: H(P,Q) = H(P) + KL(P||Q). When the true distribution P is fixed, the entropy term is a constant, so minimising cross-entropy and minimising KL give the same model, but the two numbers are not the same quantity.
Why is KL divergence not a distance?
It is asymmetric, so KL(P||Q) generally differs from KL(Q||P), and it does not satisfy the triangle inequality. It is non-negative and zero only when the distributions match, which makes it a divergence rather than a metric. Jensen-Shannon divergence is the bounded, symmetric alternative, and its square root is a true metric.
Which direction of KL divergence does variational inference minimize?
Variational inference minimises KL(q||p), where q is the approximation and p is the posterior. That direction is mode-seeking: it lets q ignore regions where p has little mass, so the approximation tends to lock onto one mode and underestimate uncertainty. Minimising KL(p||q) instead would be mass-covering and would spread the approximation across modes.
Why does mutual information beat correlation for feature selection?
Pearson correlation only detects linear relationships, while mutual information detects any statistical dependence. A U-shaped feature can have near-zero correlation and high mutual information. The trade-off is that mutual information is harder to estimate, is not adjusted for feature cardinality, and needs discrete features declared explicitly.
Why is my PyTorch cross-entropy different from the textbook value?
PyTorch uses the natural logarithm, so its loss is in nats. Textbooks usually compute entropy in bits. One nat equals about 1.4427 bits, so multiply the PyTorch value by 1.4427 to compare, or divide the textbook value by the same factor.
Sources
[1]MacKay, Information Theory, Inference, and Learning Algorithms — entropy, Gibbs' inequality, KL divergence, Jensen-Shannon, and the coding interpretation, plus the variational chapters: inference.org.uk/itila[2]Goodfellow, Bengio, and Courville, Deep Learning, chapter 3 — information theory for machine learning, including KL divergence, cross-entropy, and mutual information: deeplearningbook.org[3]scikit-learn, DecisionTreeClassifier — thecriterionparameter and the Gini default: scikit-learn.org[4]scikit-learn, mutual_info_classif — the k-NN estimator of Kraskov, Stögbauer, and Grassberger, plusdiscrete_featuresandn_neighbors: scikit-learn.org[5]SciPy, scipy.stats.entropy — entropy and KL divergence, input normalisation, and thebaseparameter: docs.scipy.org
