Sam Austinon November 13, 202613 min read

Entropy, KL Divergence, and Mutual Information Explained for ML

Entropy, KL Divergence, and Mutual Information Explained for ML
Contents

Figure 1: Entropy, KL Divergence, and Mutual Information Explained for ML — a fair die is the cleanest picture of entropy: six equally likely outcomes, log₂6 ≈ 2.585 bits of uncertainty, and no way to compress it

You call CrossEntropyLoss every day, you have seen "KL divergence" in a VAE paper, and you have probably used "mutual information" in a feature selection tutorial. Do those three ideas connect? They do, and once you see how, a lot of machine learning stops feeling like a pile of unrelated formulas.

All three answer one question: how surprised should you feel? Entropy measures the surprise in the truth. Cross-entropy measures the surprise you get when you describe the truth with the wrong model. KL divergence is the difference. Mutual information is the surprise that disappears when you learn something about another variable.

The One Question Behind All Three

Start with a single event. If it has probability p, the surprise it carries is:

−log p

A certain event carries zero surprise. A rare event carries a lot. Every quantity in this article is an average of that one term, weighted by how often each outcome occurs.

Quantity What it averages Formula
Entropy surprise under the true distribution H(P) = −Σ P log P
Cross-entropy surprise under the model, weighted by the truth H(P,Q) = −Σ P log Q
KL divergence the excess surprise KL(P‖Q) = Σ P log(P/Q)
Mutual information surprise removed by knowing X `I(X;Y) = H(Y) − H(Y

That is the whole article in one table. The rest is what each row means, when it breaks, and how to compute it without tripping over log bases.

Entropy: Measuring Uncertainty

Entropy is the average surprise of a distribution:

H(P) = − Σₓ P(x) · log P(x)

A rare event surprises you a lot, so it contributes a large amount of information; a common event barely registers. Entropy weights each surprise by how often it happens, so it is an expected value over the distribution, not a property of any single outcome.

The Twenty Questions framing is exact, not a metaphor. If outcomes are uniform over 2ᵏ possibilities, the optimal yes/no strategy needs exactly k questions, and the entropy is k bits. Entropy is the average number of yes/no answers you cannot avoid.

Three anchors worth memorising:

Distribution Entropy (bits) Entropy (nats)
Always heads (p = 1) 0 0
Fair coin (p = 0.5) 1 0.693
Biased coin (p = 0.9) 0.469 0.325

A fair coin is the maximum for two outcomes. A coin that always lands heads carries zero bits: you never feel surprised. And a uniform distribution maximises entropy over a fixed finite set of outcomes — total uncertainty means you cannot predict anything, and no other distribution on the same support is harder to guess.

That qualifier matters. The maximum-entropy statement is about a fixed finite support. For continuous variables the analogue is differential entropy, which can be negative and depends on scale — it is not a count of bits.

Bits vs Nats

The log base is a unit choice, nothing more. Base 2 gives bits, base e gives nats, and:

1 nat = 1 / ln 2 ≈ 1.4427 bits

So a value in bits is about 44% larger than the same value in nats. That is a multiplicative conversion, not an additive difference: 0.693 nats = 1 bit, and 1 nat = 1.44 bits.

PyTorch and TensorFlow use natural logs, which is why a loss value from your framework will not match a textbook example computed in bits. Multiply nats by 1.4427 to compare with the textbook, or divide bits by the same number. For language models, perplexity is exp(cross-entropy) when the loss is in nats — another place the base silently matters.

Cross-Entropy: The Cost of Using the Wrong Model

Now suppose the world follows P, but your model believes Q:

H(P, Q) = − Σₓ P(x) · log Q(x)

The coding reading is the cleanest one. An optimal code for Q assigns the event x a length of −log Q(x). If the data actually comes from P, the expected code length is H(P,Q) — the average number of units you spend describing reality with a code built for a belief that is not reality.

If Q = P, cross-entropy equals entropy and you pay the minimum possible cost. If Q is wrong, you pay extra, and the extra is exactly the KL divergence.

Cross-entropy is not symmetric either. H(P,Q) ≠ H(Q,P) in general, for the same reason KL is not.

Why It's Your Classification Loss

For a classifier, the true label is one-hot. P puts all its mass on one class y, so the sum collapses to a single term:

H(P, Q) = − log q_y

That is the whole loss. Give the correct class probability 0.9 and you pay 0.105 nats; give it 0.01 and you pay 4.605 nats. The penalty is unbounded as the predicted probability goes to zero, which is why a confidently wrong prediction dominates the gradient.

Minimising cross-entropy is maximising the likelihood of the training data. For one-hot labels, −log q_y is the negative log-likelihood of the observed class, so the classifier has been doing maximum likelihood all along — I went through that equivalence in maximum likelihood estimation [2].

Binary cross-entropy and logistic loss are the same quantity written two ways. And CrossEntropyLoss in PyTorch expects logits, not probabilities — it applies log-softmax internally, which is numerically safer than computing softmax and then taking a log.

KL Divergence: The Extra Cost

Subtract the entropy from the cross-entropy and the excess is the KL divergence:

KL(P ‖ Q) = Σₓ P(x) · log [ P(x) / Q(x) ]

The identity that makes cross-entropy make sense is:

H(P, Q) = H(P) + KL(P ‖ Q)

One line of algebra proves it:

−Σ P log Q = −Σ P log P + Σ P log(P/Q)

Entropy is the unavoidable cost of real uncertainty; KL is the extra cost of using the wrong model. When the training labels are fixed, H(P) does not depend on the model, so minimising cross-entropy is exactly minimising KL(P‖Q) [2]. Every classifier trained with cross-entropy is secretly pushing its predicted distribution toward the empirical one.

Properties You Should Know

  • Non-negative. KL(P‖Q) ≥ 0 always — Gibbs' inequality [1].
  • Zero only when identical. KL(P‖Q) = 0 exactly when P = Q almost everywhere.
  • Asymmetric. KL(P‖Q) ≠ KL(Q‖P) in general.
  • Not a distance. It fails symmetry and the triangle inequality, which is why it is called a divergence.
  • Can be infinite. If Q(x) = 0 where P(x) > 0, the log ratio blows up.
  • Not bounded above. Even between distributions on the same finite set, KL can be arbitrarily large.

If you need a symmetric, bounded alternative, Jensen-Shannon divergence is the standard one:

JSD(P ‖ Q) = ½ KL(P ‖ M) + ½ KL(Q ‖ M), with M = ½(P + Q)

JSD is bounded between 0 and log 2 in nats — between 0 and 1 in bits [1]. It is not itself a metric, but its square root is, which is the version to use if you need the triangle inequality.

Why the Direction Matters

The asymmetry is not a technicality. It changes what a fit does.

  • KL(P‖Q) penalises Q for putting low probability where P has mass. To avoid the penalty, Q must cover every mode of P. This is mass-covering, zero-avoiding, and mean-seeking — the fit spreads out and blurs across modes.
  • KL(Q‖P) penalises Q for putting mass where P has little. Q can safely ignore a mode of P as long as it fits one of them well. This is zero-forcing and mode-seeking — the fit locks onto a single mode and looks too narrow.

Variational inference minimises the second form, KL(q‖p) with q the approximation and p the posterior [1]. That is exactly why VI approximations are famously overconfident and too narrow: the objective rewards picking a mode over covering the distribution. If you have seen the KL penalty in RLHF, the direction and estimator details are a separate story — KL divergence in RLHF covers that end.

A naming warning: "forward" and "reverse" KL are used inconsistently across fields. Read the arguments, not the label.

Where KL Shows Up in ML

  • Variational autoencoders. The loss is a reconstruction term plus KL(q(z|x) ‖ p(z)), which pulls the encoder's posterior toward the N(0, I) prior. That KL term is what makes the latent space smooth enough to sample from.
  • Variational inference. The ELBO is log p(x) − KL(q(z)‖p(z|x)), so maximising the ELBO is minimising the KL to the posterior. Same identity, different clothes [2].
  • Knowledge distillation. The student minimises KL(p_teacher ‖ p_student) at a temperature, which is why soft targets carry more signal than hard labels.
  • Reinforcement learning. TRPO constrains the KL between the old and new policy; PPO approximates the same idea with a clipped ratio; RLHF adds an explicit KL penalty to a reference model.
  • Regularisation. A KL term between the posterior and a prior is a soft constraint, and a Gaussian prior recovers L2 — the same connection as MLE versus MAP.

Sometimes you want KL small, sometimes you read it as a signal. In some settings a large KL means surprise, disagreement, or information gained — but the number is only meaningful next to the distributions it was computed from.

Mutual Information: What One Variable Tells You About Another

Mutual information asks how much knowing X reduces your uncertainty about Y:

I(X; Y) = H(Y) − H(Y | X) = H(X) + H(Y) − H(X, Y)

The conditional entropy H(Y|X) is the uncertainty left in Y after you observe X. If X tells you nothing, H(Y|X) = H(Y) and the mutual information is zero. If X determines Y perfectly, H(Y|X) = 0 and the mutual information equals H(Y).

The KL reading is the one that generalises:

I(X; Y) = KL( P(X, Y) ‖ P(X) P(Y) )

MI is the divergence between the true joint distribution and the product of the marginals. Independence is exactly the product distribution, so MI measures how far your variables sit from independent [2]. It inherits non-negativity from KL, and it is symmetric because the joint and the product of marginals are.

Useful consequences:

  • I(X;Y) ≥ 0, with equality exactly when X and Y are independent.
  • It is symmetric: I(X;Y) = I(Y;X), so it cannot tell you direction or causation.
  • For discrete variables, I(X;Y) ≤ min(H(X), H(Y)) — shared information cannot exceed either variable's uncertainty.
  • The chain rule: H(X,Y) = H(X) + H(Y|X).

One caveat that bites in practice: for continuous variables, entropy is differential entropy and can be negative, but mutual information is still non-negative. The "bits" interpretation does not carry over cleanly; the estimator does.

Why MI Beats Correlation for Feature Selection

Pearson correlation measures linear association only. MI measures any statistical dependence. A feature with a U-shaped relationship to the target can have correlation near zero and mutual information well above zero — the feature is informative, but its effect cancels in a straight-line fit.

That is the pitch for mutual_info_classif and mutual_info_regression. Rank features by MI with the target, keep the top ones, and you catch relationships a correlation matrix would drop.

But MI is not a free lunch:

  1. It is not adjusted for cardinality. A high-cardinality identifier-like feature can score high by memorising noise; MI has no built-in complexity penalty.
  2. The continuous estimator is a k-nearest-neighbour estimate, with n_neighbors=3 by default — higher values reduce variance and add bias [4].
  3. discrete_features="auto" treats dense input as continuous. Integer-coded categorical columns will be fed to the continuous estimator unless you pass a boolean mask [4].
  4. The values are not comparable across datasets or sample sizes. Treat them as rankings within one run, not as absolute information.

MI also cannot tell you direction. It is symmetric, so it can say "these are dependent" but never "X causes Y". For a broader comparison of selection strategies, see feature selection: filter, wrapper, embedded.

Entropy in Decision Trees

A decision tree split is an information-gain calculation. At each node it picks the feature that reduces label uncertainty the most:

IG(Y, X) = H(Y) − H(Y | X) = I(Y; X)

That is mutual information between the feature and the label, in disguise. The greedy split search evaluates candidate thresholds, computes the weighted child entropies, and keeps the split with the largest gain.

Two details the textbooks blur:

  • ID3 and C4.5 use entropy; CART defaults to Gini impurity. scikit-learn's DecisionTreeClassifier uses criterion="gini" by default — pass criterion="entropy" if you want the information-gain version [3]. Gini is cheaper (no logarithm) and usually behaves similarly, but it is a different impurity.
  • The search is greedy and local. A split that maximises immediate information gain can be globally suboptimal; that is why ensembles and pruning exist.

All Three in Code

import numpy as np
from scipy.stats import entropy
from sklearn.feature_selection import mutual_info_classif

p = np.array([0.7, 0.2, 0.1])   # "true" distribution
q = np.array([0.5, 0.3, 0.2])   # model's distribution

H_p  = entropy(p)               # nats by default
H_pq = -(p * np.log(q)).sum()   # cross-entropy, nats
kl   = entropy(p, q)            # KL(P || Q), nats

print(H_p, H_pq, kl)
print(np.isclose(H_pq, H_p + kl))   # True
print(entropy(q, p))                 # different: KL is asymmetric

# The same numbers in bits
print(entropy(p, base=2), entropy(p, q, base=2))

# Mutual information: nonlinear dependence that correlation misses
rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 2000)
y = (x**2 + 0.05 * rng.normal(size=2000) > 0.3).astype(int)
print(np.corrcoef(x, y)[0, 1])            # near zero
print(mutual_info_classif(x.reshape(-1, 1), y, random_state=0))  # clearly positive

Run it and check four things.

  1. The identity holds numerically: H_pq equals H_p + kl to machine precision.
  2. Swapping the arguments changes the answer, which is the asymmetry in one line.
  3. The base changes the units, not the ordering. base=2 multiplies every value by 1.4427.
  4. Correlation misses the U-shape while MI catches it.

Two API details worth knowing. scipy.stats.entropy normalises its inputs if they do not sum to 1 [5] — convenient, but it will silently hide a bug where you forgot to normalise. And mutual_info_classif adds small noise to continuous features to break ties, which is what random_state controls [4].

The Cheat Sheet

  • Entropy: how uncertain is the truth? H(P) = −Σ P log P
  • Cross-entropy: what does it cost to describe the truth with my model? H(P,Q) = −Σ P log Q
  • KL divergence: how much extra do I pay for the wrong model? KL(P‖Q) = H(P,Q) − H(P)
  • Mutual information: how much does X tell me about Y? I(X;Y) = H(Y) − H(Y|X)
  • Conditional entropy: how much uncertainty remains in Y after I see X? H(Y|X) = H(X,Y) − H(X)

Common Pitfalls

  • Mixing log bases. A value in bits is 1.4427× the same value in nats — a 44% increase. Pick one base and stay in it.
  • Treating KL as a distance. It is asymmetric and violates the triangle inequality; the order of arguments is part of the meaning.
  • Forgetting zero probabilities. A zero in Q where P has mass makes KL infinite. Smooth or clip, and check for inf before you log a metric.
  • Feeding logits into a probability formula. Apply softmax first, or use a loss that does it internally — CrossEntropyLoss expects logits, NLLLoss expects log-probabilities.
  • Trusting MI estimates on small continuous samples. The k-NN estimator is noisy; treat the values as rankings, not measurements.
  • Leaving discrete_features="auto" on dense categorical data. Integer codes get the continuous estimator and the scores become meaningless.
  • Assuming MI shows direction. It is symmetric; it cannot tell you whether X causes Y.
  • Calling entropy(p, q) and assuming it normalises the way you meant. It normalises each input separately, which can mask a preprocessing bug.
  • Assuming scikit-learn's trees use entropy. The default is Gini; information gain needs criterion="entropy".

Best Book to Read

CoverBookDescriptionGet it
Cover of “Information Theory, Inference, and Learning Algorithms” Information Theory, Inference, and Learning Algorithmsby David J. C. MacKay entropy and KL from first principles, then the inference and machine-learning chapters that use them. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read chapters 1 and 2 for the coding interpretation, then chapter 4 for the KL divergence and its role in inference. MacKay's derivations are unusually geometric, and the book is free on his site if you want to try before buying.

Wrapping Up

Entropy measures uncertainty. Cross-entropy measures the cost of describing reality with an imperfect model. KL divergence is the gap between them, and that gap is why cross-entropy loss works. Mutual information measures shared information and gives you a dependence measure that correlation cannot provide.

The identity to remember is H(P,Q) = H(P) + KL(P‖Q). Everything else follows: fixed labels make the entropy term constant, so training on cross-entropy is training on KL; and mutual information is the same divergence applied to a joint distribution versus its marginals.

The experiment worth running: take a classifier you trained recently, compute its average cross-entropy, the entropy of its label distribution, and the difference. That difference is the model's KL divergence from the empirical truth, and watching it shrink as training proceeds makes the theory concrete.

If the probability foundations still feel shaky, math for machine learning maps what actually matters, and MLE versus MAP shows how the same log-probability reasoning turns into a regulariser.

Now go measure something.

Frequently Asked Questions

Is cross-entropy the same as KL divergence?

No. Cross-entropy equals entropy plus KL divergence: H(P,Q) = H(P) + KL(P||Q). When the true distribution P is fixed, the entropy term is a constant, so minimising cross-entropy and minimising KL give the same model, but the two numbers are not the same quantity.

Why is KL divergence not a distance?

It is asymmetric, so KL(P||Q) generally differs from KL(Q||P), and it does not satisfy the triangle inequality. It is non-negative and zero only when the distributions match, which makes it a divergence rather than a metric. Jensen-Shannon divergence is the bounded, symmetric alternative, and its square root is a true metric.

Which direction of KL divergence does variational inference minimize?

Variational inference minimises KL(q||p), where q is the approximation and p is the posterior. That direction is mode-seeking: it lets q ignore regions where p has little mass, so the approximation tends to lock onto one mode and underestimate uncertainty. Minimising KL(p||q) instead would be mass-covering and would spread the approximation across modes.

Why does mutual information beat correlation for feature selection?

Pearson correlation only detects linear relationships, while mutual information detects any statistical dependence. A U-shaped feature can have near-zero correlation and high mutual information. The trade-off is that mutual information is harder to estimate, is not adjusted for feature cardinality, and needs discrete features declared explicitly.

Why is my PyTorch cross-entropy different from the textbook value?

PyTorch uses the natural logarithm, so its loss is in nats. Textbooks usually compute entropy in bits. One nat equals about 1.4427 bits, so multiply the PyTorch value by 1.4427 to compare, or divide the textbook value by the same factor.

Sources

  • [1] MacKay, Information Theory, Inference, and Learning Algorithms — entropy, Gibbs' inequality, KL divergence, Jensen-Shannon, and the coding interpretation, plus the variational chapters: inference.org.uk/itila
  • [2] Goodfellow, Bengio, and Courville, Deep Learning, chapter 3 — information theory for machine learning, including KL divergence, cross-entropy, and mutual information: deeplearningbook.org
  • [3] scikit-learn, DecisionTreeClassifier — the criterion parameter and the Gini default: scikit-learn.org
  • [4] scikit-learn, mutual_info_classif — the k-NN estimator of Kraskov, Stögbauer, and Grassberger, plus discrete_features and n_neighbors: scikit-learn.org
  • [5] SciPy, scipy.stats.entropy — entropy and KL divergence, input normalisation, and the base parameter: docs.scipy.org

What are You Looking For?

esc