Sam Austinon November 13, 202613 min read

SVD Explained for Machine Learning (and Why It Powers PCA and Recommenders)

SVD Explained for Machine Learning (and Why It Powers PCA and Recommenders)
Contents

Figure 1: SVD Explained for Machine Learning (and Why It Powers PCA and Recommenders) — any matrix is a rotation, a stretch, and another rotation, and the stretch factors are the whole story

You hear "SVD" in PCA tutorials, recommender system talks, and NLP lectures. Everyone treats it as common knowledge, and almost nobody explains what it actually does.

The idea is genuinely simple. Any matrix, of any shape, is a rotation followed by a stretch followed by another rotation. Everything else — PCA, latent factors, compression, pseudoinverses — is a consequence of that one fact.

What SVD Actually Does

The decomposition is:

A = U Σ Vᵀ

For an m × n matrix A:

  • U is m × m and orthogonal — its columns are the left singular vectors.
  • Σ is m × n and diagonal — its entries are the singular values, sorted largest to smallest.
  • V is n × n and orthogonal — its columns are the right singular vectors.

Read it as three operations on a vector:

  1. Vᵀ rotates the input into a coordinate system where the matrix does something simple.
  2. Σ stretches or squashes along each of those axes, by σ₁, σ₂, ….
  3. U rotates the result into the output space.

That is what a matrix is, geometrically. A linear map takes a sphere, turns it into an ellipsoid, and the singular values are the lengths of that ellipsoid's axes.

And it works for any matrix. Any shape, any rank, no symmetry, no squareness. Eigendecomposition needs a square matrix and can fail on defective ones; SVD always exists [1].

One practical detail: you almost never want the full U and V when m or n is large. np.linalg.svd(A, full_matrices=False) returns the thin version, where U is m × r and V is n × r with r = min(m, n). That is what everyone means in practice.

The Layered View

There is a second way to read the same formula that is more useful for machine learning. Expand the product:

A = σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ + … + σᵣuᵣvᵣᵀ

Each term is a rank-1 matrix — an outer product of one left vector and one right vector. The full matrix is a sum of r such layers, where r is the rank, and the singular values say how much each layer contributes.

Think of a photo built from layers. The first layer paints the big shapes, the next adds structure, and the last layers add grain that you would not miss.

That image is the whole basis of compression, denoising, and latent factor models. If the tail layers are small, you can delete them.

Truncated SVD: The Best Low-Rank Approximation

Keep the top k layers and drop the rest:

A_k = σ₁u₁v₁ᵀ + … + σₖuₖvₖᵀ

You do not just get a decent approximation. You get the optimal one. The Eckart-Young-Mirsky theorem states that this truncated SVD is the closest rank-k matrix to A in any unitarily invariant norm — Frobenius, spectral, nuclear [1]. No other rank-k matrix is closer.

The error is predictable, and the exact form depends on the norm:

  • Frobenius: ‖A − A_k‖_F² = σₖ₊₁² + σₖ₊₂² + … + σᵣ²
  • Spectral: ‖A − A_k‖₂ = σₖ₊₁

So each discarded singular value contributes its square to the squared Frobenius error. If the tail is tiny, you lose almost nothing — and you can compute the loss before you commit to a k.

This is why truncation is not a heuristic. It is a projection onto the best possible subspace of that dimension, and the theorem is what separates "SVD is useful" from "SVD is optimal".

Why SVD Is Numerically Better Than Eigendecomposition

PCA is often taught as an eigenproblem on the covariance matrix. That version works in exact arithmetic and is a bad idea in floating point.

Forming XᵀX squares the condition number. If your data matrix has condition number κ, the covariance has κ², and in float64 you lose about log₁₀(κ²) digits before you start. SVD works on X directly, so it only ever sees κ. That is the same argument that makes lstsq better than the normal equation, which I went through in normal equation vs gradient descent.

SVD is also backward stable, meaning the computed factors are the exact factors of a slightly perturbed matrix. Eigendecomposition of an ill-conditioned matrix has no such guarantee.

Two ambiguities worth knowing before you compare two runs:

  • Sign. uᵢ and −uᵢ give the same outer product, so each component is defined only up to a sign. Two libraries can return opposite signs for the same component, and both are correct.
  • Repeated singular values. When σᵢ = σᵢ₊₁, the individual vectors are not unique — only the subspace they span is. Any rotation within that subspace is a valid answer.

Neither is a bug. They are why you compare subspaces, not vectors, when you check an implementation.

Why SVD Powers PCA

Take your data matrix X, n rows and d columns, and centre each column. Then:

X = U Σ Vᵀ

and the mapping to PCA is exact:

SVD piece PCA meaning
Columns of V Principal directions (loadings)
σᵢ² / (n − 1) Variance explained by component i
U Σ Scores — the data in the new basis

That is not an analogy. PCA is defined as projecting onto the directions of maximum variance, and the right singular vectors are exactly those directions [5]. The covariance-eigenvector route reaches the same answer by a longer and less stable road [4].

Two conditions matter:

  1. Centre first. Without centring, the first component chases the mean instead of the variance, and the equivalence breaks.
  2. Scaling is a separate decision. Centring is required for the algebra; dividing by the standard deviation is a modelling choice, and it changes what "variance" means.

For a square, symmetric, positive semi-definite matrix the two decompositions coincide — which is why the covariance matrix is the one place where the eigen-story is exactly right. For a symmetric matrix that is not positive semi-definite, the singular values are the absolute eigenvalues, so the signs are lost. That is a real difference, not a technicality.

And yes, scikit-learn's PCA uses SVD internally — full, randomized, or arpack depending on the solver you pick [4]. The eigen-decomposition of the covariance is available in textbooks, not in the default code path.

Why SVD Powers Recommenders

Imagine a ratings matrix R: users down the rows, movies across the columns, and blanks wherever nobody watched.

The assumption that makes it tractable is low rank. Users do not have independent tastes. A handful of latent factors — action versus drama, old versus new, mainstream versus obscure — explain most of the variation. So the enormous sparse matrix is really a small number of patterns plus noise.

Factor it into user and item vectors:

R ≈ P Qᵀ, and a predicted rating is r̂ᵤᵢ = pᵤ · qᵢ

Users with similar vectors like similar things, and the dot product is the similarity score. The Netflix Prize, which ran from 2006 to 2009, is what made this approach famous [3].

The catch: the matrix is almost entirely empty

The Netflix dataset had about 100 million ratings across roughly 480,000 users and 17,770 movies. That is a fill rate of about 1.2% — 98.8% of the matrix is missing.

Classic SVD requires a fully observed matrix. You cannot decompose what you do not have, and the usual workaround — fill the blanks with the mean, then run truncated SVD — teaches the model that every user gave an average rating to everything they never watched. The imputed values are treated as evidence, and they dominate, because they outnumber the real ones by eighty to one.

The fix is to optimise only over the observed entries:

min Σ_{(u,i) observed} (rᵤᵢ − pᵤ·qᵢ)² + λ(‖pᵤ‖² + ‖qᵢ‖²)

That objective is not an SVD. It is a regularised least-squares problem solved by SGD or alternating least squares, and the regularisation term is not optional — with 1.2% density, an unregularised model memorises the observed ratings and predicts noise everywhere else.

Simon Funk's 2006 approach was exactly this: an iterative SGD fit on the observed entries. The recommender literature still calls the family "SVD", but the algorithms are approximations, not decompositions [3].

Why the low-rank idea is principled

Minimising the rank directly is NP-hard. The convex surrogate is the nuclear norm — the sum of the singular values — and Candès and Recht showed that under incoherence conditions, minimising it recovers the true low-rank matrix from a small sample of entries [2]. That result is why matrix completion is not just a heuristic that happens to work.

Modern systems still build on this. The embedding layers in a deep recommender are a learned factorisation with the same structure; the difference is that the interaction is a neural network instead of a dot product.

Other Places SVD Shows Up

  • Latent semantic analysis. SVD on a term-document matrix puts words and documents in a shared concept space, which is what makes retrieval work when the query and the document share no words.
  • Least squares and the pseudoinverse. The Moore-Penrose pseudoinverse is V Σ⁺ Uᵀ, and it solves overdetermined systems and rank-deficient ones in one formula.
  • Denoising. Small singular values often capture noise, so dropping them is a principled filter — the same truncation, justified by the error formula above.
  • Compression. Keep the top k layers and store k(m + n + 1) numbers instead of mn.
  • Condition number estimation. σ₁/σᵣ is the 2-norm condition number, which is why SVD is the diagnostic you reach for when a solve looks wrong.

SVD in Code

import numpy as np
from sklearn.decomposition import TruncatedSVD

rng = np.random.default_rng(0)
A = rng.normal(size=(100, 5)) @ rng.normal(size=(5, 40))   # rank 5
A += 0.05 * rng.normal(size=A.shape)                        # add noise

U, s, Vt = np.linalg.svd(A, full_matrices=False)
print(s[:8].round(2))        # five big values, then a sharp drop

k = 5
A_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]
err = np.linalg.norm(A - A_k) / np.linalg.norm(A)
print(f"relative error with k={k}: {err:.4f}")

# The same idea with scikit-learn, which also accepts sparse input
svd = TruncatedSVD(n_components=5, random_state=0)
Z = svd.fit_transform(A)
print(svd.explained_variance_ratio_.sum())

Run it and watch the singular values. The cliff after the fifth one is the signal: five real patterns, then noise. That cliff is how you spot the rank, and it is visible before you fit anything.

One detail that trips people up: TruncatedSVD does not centre your data. That is deliberate — centring destroys sparsity — but it means TruncatedSVD is not PCA. For PCA, use PCA or centre the data yourself.

For matrices too large for a full decomposition, use a randomised SVD — Halko, Martinsson, and Tropp's algorithm computes the top k components in roughly O(mnk) instead of O(mn·min(m,n)), with a small approximation error you can control [1]. It is what PCA(svd_solver="randomized") does under the hood.

How to Choose k

  • Look for the cliff. Plot the singular values on a log scale and find where they fall off.
  • Set a variance target. Keep enough components to cover 90–95% of the total squared singular values.
  • Validate on your task. For recommenders, hold out ratings and pick the k with the lowest held-out error.
  • Start small. A few dozen factors often suffice, and more will overfit.

Option three matters most, because the right rank depends on what you do with the result. And high explained variance is not the same as usefulness — I went through that trap in the PCA tutorial, where a component with real variance turned out to hurt the classifier.

Common Mistakes

  • Forgetting to centre before PCA. Your components will chase the mean instead of the variance.
  • Treating missing ratings as zeros. The model learns that users hate everything they have not seen.
  • Mean-imputing a sparse matrix and calling it SVD. The imputed cells outnumber the real ones and dominate the fit.
  • Running a full SVD on a huge matrix. Use a truncated or randomised solver; the full one is cubic in the smaller dimension.
  • Picking k by habit. Too few factors underfit, too many memorise noise.
  • Skipping regularisation in recommenders. At 1% density, an unregularised factorisation is a lookup table.
  • Comparing singular vectors across runs. Signs flip, and repeated singular values make the vectors non-unique.
  • Confusing TruncatedSVD with PCA. One centres, the other does not, and the difference is a mean-sized bias in every component.

Best Book to Read

CoverBookDescriptionGet it
Cover of “Mathematics for Machine Learning” Mathematics for Machine Learningby Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong chapter 4 builds the SVD from the geometry up, and chapter 10 applies it to PCA and latent variable models. View on Amazon

As an Amazon Associate I earn from qualifying purchases. Links on this page are affiliate links, and I may earn a commission at no extra cost to you.

Read chapter 4 with a pencil and sketch the unit circle turning into an ellipse. That one drawing makes the three-factor reading permanent, and the rest of the book assumes you have it. The PDF is free at mml-book.github.io.

Wrapping Up

SVD factors any matrix into a rotation, a stretch, and a rotation. The stretch factors are the singular values, and they are ordered by how much each direction matters.

Truncating the tail gives the provably best low-rank approximation — Eckart-Young-Mirsky, not a heuristic — and the error is exactly the sum of the squared discarded values.

That single property explains PCA (the right singular vectors are the principal directions), recommenders (the latent factors are a low-rank fit, with the missing-data problem solved by optimising over observed entries only), LSA, compression, denoising, and the pseudoinverse.

The practical rule: use SVD when the matrix is complete, use a regularised factorisation when it is not, and never form XᵀX when you can decompose X directly.

Now go decompose something and look for the cliff.

Frequently Asked Questions

What does SVD actually do in plain terms?

It factors any matrix into a rotation of the input space, a stretch along each axis, and a rotation into the output space. The stretch factors are the singular values, sorted from largest to smallest, and they tell you how much each direction matters.

Why is truncated SVD the best low-rank approximation?

The Eckart-Young-Mirsky theorem proves that dropping the smallest singular values gives the closest rank-k matrix in any unitarily invariant norm. No other rank-k matrix is closer, which is why truncation is not a heuristic but an optimal projection.

Is PCA the same as SVD?

PCA on centred data is the SVD of the centred data matrix. The right singular vectors are the principal directions, the squared singular values divided by n minus one are the variances, and U times Sigma gives the scores. The covariance-eigenvector version is the same computation done less stably.

Why do recommender systems use SVD if ratings are mostly missing?

Classic SVD needs a fully observed matrix, so recommender papers use a regularised factorisation fitted only on observed entries, usually with SGD or alternating least squares. It is often called SVD in the literature, but it is a different optimisation problem.

Why do singular vectors flip sign between runs?

Because u and minus u give the same outer product, so the decomposition is only defined up to a sign per component. When singular values repeat, even the individual directions are not unique, only the subspace they span.

Sources

  • [1] Halko, Martinsson, and Tropp, Finding Structure with Randomness (2011) — the randomised SVD algorithm, with the Eckart-Young theorem stated and used throughout: arxiv.org/abs/0909.4061
  • [2] Candès and Recht, Exact Matrix Completion via Convex Optimization (2009) — the nuclear-norm argument that justifies low-rank recovery from partial observations: arxiv.org/abs/0805.4471
  • [3] Koren, Bell, and Volinsky, Matrix Factorization Techniques for Recommender Systems (2009) — the Netflix Prize lessons on SGD, ALS, and regularisation: doi.org/10.1109/MC.2009.263
  • [4] scikit-learn, PCA — the SVD-based solvers and the explained_variance_ratio_ definition: scikit-learn.org
  • [5] Deisenroth, Faisal, and Ong, Mathematics for Machine Learning — chapter 4 derives the SVD geometrically; chapter 10 covers PCA and latent variable models: mml-book.github.io

What are You Looking For?

esc