Contents
Figure 1: SVD Explained for Machine Learning (and Why It Powers PCA and Recommenders) — any matrix is a rotation, a stretch, and another rotation, and the stretch factors are the whole story
You hear "SVD" in PCA tutorials, recommender system talks, and NLP lectures. Everyone treats it as common knowledge, and almost nobody explains what it actually does.
The idea is genuinely simple. Any matrix, of any shape, is a rotation followed by a stretch followed by another rotation. Everything else — PCA, latent factors, compression, pseudoinverses — is a consequence of that one fact.
What SVD Actually Does
The decomposition is:
A = U Σ Vᵀ
For an m × n matrix A:
Uism × mand orthogonal — its columns are the left singular vectors.Σism × nand diagonal — its entries are the singular values, sorted largest to smallest.Visn × nand orthogonal — its columns are the right singular vectors.
Read it as three operations on a vector:
Vᵀrotates the input into a coordinate system where the matrix does something simple.Σstretches or squashes along each of those axes, byσ₁, σ₂, ….Urotates the result into the output space.
That is what a matrix is, geometrically. A linear map takes a sphere, turns it into an ellipsoid, and the singular values are the lengths of that ellipsoid's axes.
And it works for any matrix. Any shape, any rank, no symmetry, no squareness. Eigendecomposition needs a square matrix and can fail on defective ones; SVD always exists [1].
One practical detail: you almost never want the full U and V when m or n is large. np.linalg.svd(A, full_matrices=False) returns the thin version, where U is m × r and V is n × r with r = min(m, n). That is what everyone means in practice.
The Layered View
There is a second way to read the same formula that is more useful for machine learning. Expand the product:
A = σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ + … + σᵣuᵣvᵣᵀ
Each term is a rank-1 matrix — an outer product of one left vector and one right vector. The full matrix is a sum of r such layers, where r is the rank, and the singular values say how much each layer contributes.
Think of a photo built from layers. The first layer paints the big shapes, the next adds structure, and the last layers add grain that you would not miss.
That image is the whole basis of compression, denoising, and latent factor models. If the tail layers are small, you can delete them.
Truncated SVD: The Best Low-Rank Approximation
Keep the top k layers and drop the rest:
A_k = σ₁u₁v₁ᵀ + … + σₖuₖvₖᵀ
You do not just get a decent approximation. You get the optimal one. The Eckart-Young-Mirsky theorem states that this truncated SVD is the closest rank-k matrix to A in any unitarily invariant norm — Frobenius, spectral, nuclear [1]. No other rank-k matrix is closer.
The error is predictable, and the exact form depends on the norm:
- Frobenius:
‖A − A_k‖_F² = σₖ₊₁² + σₖ₊₂² + … + σᵣ² - Spectral:
‖A − A_k‖₂ = σₖ₊₁
So each discarded singular value contributes its square to the squared Frobenius error. If the tail is tiny, you lose almost nothing — and you can compute the loss before you commit to a k.
This is why truncation is not a heuristic. It is a projection onto the best possible subspace of that dimension, and the theorem is what separates "SVD is useful" from "SVD is optimal".
Why SVD Is Numerically Better Than Eigendecomposition
PCA is often taught as an eigenproblem on the covariance matrix. That version works in exact arithmetic and is a bad idea in floating point.
Forming XᵀX squares the condition number. If your data matrix has condition number κ, the covariance has κ², and in float64 you lose about log₁₀(κ²) digits before you start. SVD works on X directly, so it only ever sees κ. That is the same argument that makes lstsq better than the normal equation, which I went through in normal equation vs gradient descent.
SVD is also backward stable, meaning the computed factors are the exact factors of a slightly perturbed matrix. Eigendecomposition of an ill-conditioned matrix has no such guarantee.
Two ambiguities worth knowing before you compare two runs:
- Sign.
uᵢand−uᵢgive the same outer product, so each component is defined only up to a sign. Two libraries can return opposite signs for the same component, and both are correct. - Repeated singular values. When
σᵢ = σᵢ₊₁, the individual vectors are not unique — only the subspace they span is. Any rotation within that subspace is a valid answer.
Neither is a bug. They are why you compare subspaces, not vectors, when you check an implementation.
Why SVD Powers PCA
Take your data matrix X, n rows and d columns, and centre each column. Then:
X = U Σ Vᵀ
and the mapping to PCA is exact:
| SVD piece | PCA meaning |
|---|---|
Columns of V |
Principal directions (loadings) |
σᵢ² / (n − 1) |
Variance explained by component i |
U Σ |
Scores — the data in the new basis |
That is not an analogy. PCA is defined as projecting onto the directions of maximum variance, and the right singular vectors are exactly those directions [5]. The covariance-eigenvector route reaches the same answer by a longer and less stable road [4].
Two conditions matter:
- Centre first. Without centring, the first component chases the mean instead of the variance, and the equivalence breaks.
- Scaling is a separate decision. Centring is required for the algebra; dividing by the standard deviation is a modelling choice, and it changes what "variance" means.
For a square, symmetric, positive semi-definite matrix the two decompositions coincide — which is why the covariance matrix is the one place where the eigen-story is exactly right. For a symmetric matrix that is not positive semi-definite, the singular values are the absolute eigenvalues, so the signs are lost. That is a real difference, not a technicality.
And yes, scikit-learn's PCA uses SVD internally — full, randomized, or arpack depending on the solver you pick [4]. The eigen-decomposition of the covariance is available in textbooks, not in the default code path.
Why SVD Powers Recommenders
Imagine a ratings matrix R: users down the rows, movies across the columns, and blanks wherever nobody watched.
The assumption that makes it tractable is low rank. Users do not have independent tastes. A handful of latent factors — action versus drama, old versus new, mainstream versus obscure — explain most of the variation. So the enormous sparse matrix is really a small number of patterns plus noise.
Factor it into user and item vectors:
R ≈ P Qᵀ, and a predicted rating is r̂ᵤᵢ = pᵤ · qᵢ
Users with similar vectors like similar things, and the dot product is the similarity score. The Netflix Prize, which ran from 2006 to 2009, is what made this approach famous [3].
The catch: the matrix is almost entirely empty
The Netflix dataset had about 100 million ratings across roughly 480,000 users and 17,770 movies. That is a fill rate of about 1.2% — 98.8% of the matrix is missing.
Classic SVD requires a fully observed matrix. You cannot decompose what you do not have, and the usual workaround — fill the blanks with the mean, then run truncated SVD — teaches the model that every user gave an average rating to everything they never watched. The imputed values are treated as evidence, and they dominate, because they outnumber the real ones by eighty to one.
The fix is to optimise only over the observed entries:
min Σ_{(u,i) observed} (rᵤᵢ − pᵤ·qᵢ)² + λ(‖pᵤ‖² + ‖qᵢ‖²)
That objective is not an SVD. It is a regularised least-squares problem solved by SGD or alternating least squares, and the regularisation term is not optional — with 1.2% density, an unregularised model memorises the observed ratings and predicts noise everywhere else.
Simon Funk's 2006 approach was exactly this: an iterative SGD fit on the observed entries. The recommender literature still calls the family "SVD", but the algorithms are approximations, not decompositions [3].
Why the low-rank idea is principled
Minimising the rank directly is NP-hard. The convex surrogate is the nuclear norm — the sum of the singular values — and Candès and Recht showed that under incoherence conditions, minimising it recovers the true low-rank matrix from a small sample of entries [2]. That result is why matrix completion is not just a heuristic that happens to work.
Modern systems still build on this. The embedding layers in a deep recommender are a learned factorisation with the same structure; the difference is that the interaction is a neural network instead of a dot product.
Other Places SVD Shows Up
- Latent semantic analysis. SVD on a term-document matrix puts words and documents in a shared concept space, which is what makes retrieval work when the query and the document share no words.
- Least squares and the pseudoinverse. The Moore-Penrose pseudoinverse is
V Σ⁺ Uᵀ, and it solves overdetermined systems and rank-deficient ones in one formula. - Denoising. Small singular values often capture noise, so dropping them is a principled filter — the same truncation, justified by the error formula above.
- Compression. Keep the top
klayers and storek(m + n + 1)numbers instead ofmn. - Condition number estimation.
σ₁/σᵣis the 2-norm condition number, which is why SVD is the diagnostic you reach for when a solve looks wrong.
SVD in Code
import numpy as np
from sklearn.decomposition import TruncatedSVD
rng = np.random.default_rng(0)
A = rng.normal(size=(100, 5)) @ rng.normal(size=(5, 40)) # rank 5
A += 0.05 * rng.normal(size=A.shape) # add noise
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print(s[:8].round(2)) # five big values, then a sharp drop
k = 5
A_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]
err = np.linalg.norm(A - A_k) / np.linalg.norm(A)
print(f"relative error with k={k}: {err:.4f}")
# The same idea with scikit-learn, which also accepts sparse input
svd = TruncatedSVD(n_components=5, random_state=0)
Z = svd.fit_transform(A)
print(svd.explained_variance_ratio_.sum())
Run it and watch the singular values. The cliff after the fifth one is the signal: five real patterns, then noise. That cliff is how you spot the rank, and it is visible before you fit anything.
One detail that trips people up: TruncatedSVD does not centre your data. That is deliberate — centring destroys sparsity — but it means TruncatedSVD is not PCA. For PCA, use PCA or centre the data yourself.
For matrices too large for a full decomposition, use a randomised SVD — Halko, Martinsson, and Tropp's algorithm computes the top k components in roughly O(mnk) instead of O(mn·min(m,n)), with a small approximation error you can control [1]. It is what PCA(svd_solver="randomized") does under the hood.
How to Choose k
- Look for the cliff. Plot the singular values on a log scale and find where they fall off.
- Set a variance target. Keep enough components to cover 90–95% of the total squared singular values.
- Validate on your task. For recommenders, hold out ratings and pick the
kwith the lowest held-out error. - Start small. A few dozen factors often suffice, and more will overfit.
Option three matters most, because the right rank depends on what you do with the result. And high explained variance is not the same as usefulness — I went through that trap in the PCA tutorial, where a component with real variance turned out to hurt the classifier.
Common Mistakes
- Forgetting to centre before PCA. Your components will chase the mean instead of the variance.
- Treating missing ratings as zeros. The model learns that users hate everything they have not seen.
- Mean-imputing a sparse matrix and calling it SVD. The imputed cells outnumber the real ones and dominate the fit.
- Running a full SVD on a huge matrix. Use a truncated or randomised solver; the full one is cubic in the smaller dimension.
- Picking
kby habit. Too few factors underfit, too many memorise noise. - Skipping regularisation in recommenders. At 1% density, an unregularised factorisation is a lookup table.
- Comparing singular vectors across runs. Signs flip, and repeated singular values make the vectors non-unique.
- Confusing
TruncatedSVDwithPCA. One centres, the other does not, and the difference is a mean-sized bias in every component.
Best Book to Read
| Cover | Book | Description | Get it |
|---|---|---|---|
![]() |
Mathematics for Machine Learning | chapter 4 builds the SVD from the geometry up, and chapter 10 applies it to PCA and latent variable models. | View on Amazon |
As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.
Read chapter 4 with a pencil and sketch the unit circle turning into an ellipse. That one drawing makes the three-factor reading permanent, and the rest of the book assumes you have it. The PDF is free at mml-book.github.io.
Wrapping Up
SVD factors any matrix into a rotation, a stretch, and a rotation. The stretch factors are the singular values, and they are ordered by how much each direction matters.
Truncating the tail gives the provably best low-rank approximation — Eckart-Young-Mirsky, not a heuristic — and the error is exactly the sum of the squared discarded values.
That single property explains PCA (the right singular vectors are the principal directions), recommenders (the latent factors are a low-rank fit, with the missing-data problem solved by optimising over observed entries only), LSA, compression, denoising, and the pseudoinverse.
The practical rule: use SVD when the matrix is complete, use a regularised factorisation when it is not, and never form XᵀX when you can decompose X directly.
Now go decompose something and look for the cliff.
Frequently Asked Questions
What does SVD actually do in plain terms?
It factors any matrix into a rotation of the input space, a stretch along each axis, and a rotation into the output space. The stretch factors are the singular values, sorted from largest to smallest, and they tell you how much each direction matters.
Why is truncated SVD the best low-rank approximation?
The Eckart-Young-Mirsky theorem proves that dropping the smallest singular values gives the closest rank-k matrix in any unitarily invariant norm. No other rank-k matrix is closer, which is why truncation is not a heuristic but an optimal projection.
Is PCA the same as SVD?
PCA on centred data is the SVD of the centred data matrix. The right singular vectors are the principal directions, the squared singular values divided by n minus one are the variances, and U times Sigma gives the scores. The covariance-eigenvector version is the same computation done less stably.
Why do recommender systems use SVD if ratings are mostly missing?
Classic SVD needs a fully observed matrix, so recommender papers use a regularised factorisation fitted only on observed entries, usually with SGD or alternating least squares. It is often called SVD in the literature, but it is a different optimisation problem.
Why do singular vectors flip sign between runs?
Because u and minus u give the same outer product, so the decomposition is only defined up to a sign per component. When singular values repeat, even the individual directions are not unique, only the subspace they span.
Sources
[1]Halko, Martinsson, and Tropp, Finding Structure with Randomness (2011) — the randomised SVD algorithm, with the Eckart-Young theorem stated and used throughout: arxiv.org/abs/0909.4061[2]Candès and Recht, Exact Matrix Completion via Convex Optimization (2009) — the nuclear-norm argument that justifies low-rank recovery from partial observations: arxiv.org/abs/0805.4471[3]Koren, Bell, and Volinsky, Matrix Factorization Techniques for Recommender Systems (2009) — the Netflix Prize lessons on SGD, ALS, and regularisation: doi.org/10.1109/MC.2009.263[4]scikit-learn, PCA — the SVD-based solvers and theexplained_variance_ratio_definition: scikit-learn.org[5]Deisenroth, Faisal, and Ong, Mathematics for Machine Learning — chapter 4 derives the SVD geometrically; chapter 10 covers PCA and latent variable models: mml-book.github.io
