K-Means Clustering Tutorial: Choosing K and Evaluating Clusters

Contents

Figure 1: The groups are real — but only if you choose k well

k-means from scratch, four ways to pick k, and the cases where it lies

K-means has one big problem: you have to tell it how many clusters to find, and the data won't tell you. Pick k = 3 and you get three clusters. Pick k = 30 and you get thirty. Both look equally confident.

So the real skill isn't running k-means. It's choosing k and judging whether the result means anything. Below, we build k-means from scratch, match it against scikit-learn, and test four ways to choose k, plus the cases where k-means is simply the wrong tool. Every number comes from a script run with scikit-learn 1.9.1.

How K-Means Works

K-means groups points around centers (centroids). The algorithm, known as Lloyd's algorithm, repeats two steps:

1. Assign: each point joins its nearest center.
2. Update: each center moves to the average of its points.

Repeat until the centers stop moving. The score it minimizes is inertia: the total squared distance from each point to its center. Lower inertia means tighter clusters.

Two catches. K-means finds a local optimum, so the starting centers matter. And it measures distance in straight lines, so scale and shape matter too. We'll test both.

K-Means From Scratch

First, smart starting centers. k-means++ picks the first center at random, then picks each later one with probability proportional to its squared distance from the nearest chosen center. Far-away points are more likely to be picked, which spreads centers out:

import numpy as np

def kmeans_pp_init(X, k, rng):
    centers = [X[rng.integers(len(X))]]
    for _ in range(1, k):
        d2 = np.min([((X - c) ** 2).sum(1) for c in centers], axis=0)
        centers.append(X[rng.choice(len(X), p=d2 / d2.sum())])
    return np.array(centers)

Then the main loop, with several restarts, keeping the lowest-inertia run:

def kmeans(X, k, n_init=10, max_iter=300, tol=1e-8, seed=0):
    rng = np.random.default_rng(seed)
    best = None
    for _ in range(n_init):
        centers = kmeans_pp_init(X, k, rng)
        for _ in range(max_iter):
            d2 = ((X[:, None, :] - centers[None]) ** 2).sum(2)
            labels = d2.argmin(1)                                  # assign
            new = np.array([X[labels == j].mean(0) if (labels == j).any()
                            else centers[j] for j in range(k)])    # update
            shift = ((new - centers) ** 2).sum()
            centers = new
            if shift <= tol:
                break
        d2 = ((X[:, None, :] - centers[None]) ** 2).sum(2)
        labels, inertia = d2.argmin(1), d2.min(1).sum()
        if best is None or inertia < best[2]:
            best = (labels, centers, inertia)
    return best

Checking Against Scikit-learn

I generated 600 points in 4 true clusters with make_blobs. Both versions found the same solution:

	Inertia	ARI vs truth
From scratch	1905.309	1.0000
scikit-learn	1905.309	1.0000

ARI (adjusted Rand index) compares your clusters to known labels. 1 means a perfect match, and values near 0 mean random. Fair warning: these blobs are easy. Real data rarely recovers a perfect score.

Choosing K: Four Methods

Here's the key table. I ran k from 1 to 10 on the blobs, where the true answer is 4:

k	Inertia	Silhouette	Calinski-Harabasz	Davies-Bouldin
1	45,298.6	n/a	n/a	n/a
2	19,083.6	0.5681	821.5	0.8035
3	6,045.7	0.7292	1,938.1	0.3747
4	1,905.3	0.7530	4,524.6	0.3500
5	1,722.4	0.6586	3,763.4	0.6419
6	1,550.7	0.5434	3,351.5	0.8524
8	1,222.4	0.3354	3,049.4	1.1562
10	972.2	0.3623	2,989.0	0.9638

The Elbow Method

Inertia always falls as k grows. More centers always fit better, so you can't pick the minimum. You look for the elbow, where the gains flatten.

The drops tell the story. Going from 3 to 4 clusters cut inertia by 4,140. From 4 to 5, it fell by only 183. That cliff is the elbow, at k = 4. On clean data it's obvious. On messy data it's often a gentle curve where reasonable people disagree.

Silhouette Score

For each point, the silhouette compares its distance to its own cluster against its distance to the nearest other cluster. It runs from -1 to 1, and higher is better. Average it over all points, and the best k is where it peaks. Here it peaks at k = 4 (0.7530).

Calinski-Harabasz and Davies-Bouldin

Two more internal scores:

• Calinski-Harabasz: the ratio of between-cluster to within-cluster spread. Higher is better. Peak: k = 4.
• Davies-Bouldin: the average similarity between each cluster and its closest rival. Lower is better. Minimum: k = 4.

All four methods agreed. That's the dream scenario, and FYI, it's rare outside tidy example data. When methods disagree, that's information too: your clusters may not be well defined.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

for k in range(2, 11):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X)
    print(k, silhouette_score(X, labels))

Evaluating Clusters on Real Data

Wine data gives us a stricter test: 178 wines, 13 chemical measurements, and three known cultivars — the same dataset used in the naive Bayes guide. I ran k-means with k = 3 twice:

Features	ARI vs cultivar	Silhouette
Raw	0.3711	0.5711
Standardized	0.8975	0.2849

Read that carefully. The raw version has the higher silhouette (0.5711), and the much worse match to the real groups (0.3711). One feature, proline, spans about 1,400 units while others span a few. It dominates distances, so the clusters follow proline alone. They look tidy by the silhouette's measure, but they don't reflect the actual wine types.

Lesson one: scale your features before k-means, as in the k-NN guide. Lesson two: internal scores like silhouette measure tightness, not truth. They can reward the wrong answer.

Notice too that the standardized silhouette is low in absolute terms, 0.2849. On standardized wine, k = 3 scored best (0.285), but k = 2 (0.268) was close. A weak silhouette means overlapping clusters, not a broken method.

Initialization Matters

K-means can land in a poor local optimum. I ran the blobs 50 times per setup, each with a different random seed:

Setup	Worst inertia	Runs near the best (of 50)
Random init, n_init=1	14,801.9	30
k-means++, n_init=1	5,891.7	49
k-means++, n_init=10	1,905.3	50

One random start failed to find the good answer in 20 of 50 runs, and its worst result had nearly eight times the best inertia. k-means++ with ten restarts found it every time. scikit-learn defaults to k-means++, so you get this protection for free. Just don't turn off the restarts to save time.

When K-Means Is the Wrong Tool

K-means assumes clusters are roughly round and similar in size. To test that, I generated two interleaved half-moons:

Method	ARI vs truth
K-means (k = 2)	0.2525
DBSCAN	1.0000

K-means cut the moons in half with a straight line. Its silhouette was still a respectable 0.4887, which shows again that a decent internal score doesn't mean a correct answer. DBSCAN, a density-based method, separated the moons perfectly. I chose its eps=0.2 setting by eye and did not tune it, but it's sensitive to that setting, so check it on your own data. For more algorithms of this kind, see the algorithms guide in this series.

A Practical Workflow

1. Scale your features (standardize them).
2. Run k over a range and plot inertia. Look for the elbow.
3. Check silhouette and at least one other score.
4. Compare against labels if you have any, using ARI.
5. Inspect the clusters. Do the groups make sense to a human? In my experience this step beats every metric.
6. Try another algorithm if clusters aren't round.

If the scores disagree or look weak, say so in your write-up. "The data doesn't have clear clusters" is a legitimate finding.

Common Pitfalls

• Running k-means on unscaled features and letting the widest-ranging variable decide everything.
• Picking k from the elbow alone when other scores disagree — report the disagreement.
• Trusting a high silhouette as proof the clusters are correct.
• Using one random initialization with n_init=1 to "save time".
• Accepting round-cluster results on non-round shapes like moons or rings.
• Skipping human inspection of the clusters.
• Reporting a weak result as a strong one instead of saying the data has no clear clusters.
• Comparing inertia across different datasets — it is only comparable within the same data.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

How do I choose k in k-means?

Use several internal scores and treat agreement as evidence: the elbow in the inertia curve (where gains flatten), the silhouette score (peak), Calinski-Harabasz (higher is better) and Davies-Bouldin (lower is better). On tidy synthetic data all four pointed at the true k, but on real data they often disagree or look weak — which is itself a finding that the clusters are not well defined. If you have labels, compare with ARI, and always inspect the clusters by eye.

Why should I scale features before k-means?

Because k-means measures straight-line distance, so the widest-ranging feature dominates the geometry. On the wine dataset, proline spans about 1,400 units while others span a few: with raw features the clusters followed proline alone (ARI 0.3711 vs the real cultivars) even though the silhouette looked excellent at 0.5711. After standardization the match to real groups jumped to 0.8975 while the silhouette dropped to 0.2849 — tidy-looking clusters were the wrong clusters.

What does the silhouette score actually measure?

Tightness and separation, not correctness. For each point it compares the distance to its own cluster against the distance to the nearest other cluster, giving a value from -1 to 1 that is averaged over all points. It can reward a bad partition: k-means cutting interleaved half-moons in half still scored a respectable 0.4887 silhouette while its ARI versus the truth was only 0.2525. Treat it as a shape diagnostic, not a truth test.

When is k-means the wrong algorithm?

When clusters are not roughly round and similar in size. K-means cuts with straight Voronoi boundaries, so interleaved or elongated shapes get split badly — it scored ARI 0.2525 on two half-moons where DBSCAN got a perfect 1.0000. For non-round clusters, varying densities, or noise-heavy data, reach for DBSCAN or another density-based method, and remember that internal scores may still look fine while the answer is wrong.

Wrapping This Up

Choosing k is judgment, supported by numbers. The elbow, silhouette, Calinski-Harabasz and Davies-Bouldin scores gave the same answer on tidy data, and the wine example shows how a good-looking score can hide a bad result. Scale first, restart often, and sanity-check the clusters yourself.

Next step: grab the script and try your own dataset. Remove the scaling, change the seed, and watch what moves.

Related Articles

• K-Nearest Neighbors Tutorial: Simple Classification in Python
• Naive Bayes Classifier: Text and Tabular Examples in Python
• Machine Learning Algorithms Explained: Complete Beginner's Guide
• Decision Trees in Python: Build, Visualize, and Prune