SVM C and Gamma Explained: Tuning the RBF Kernel

Contents

Figure 1: SVM C and Gamma Explained — two knobs that set complexity from opposite directions

You fit an RBF support vector machine, accept the defaults, and get a decent score. Then someone asks you to improve it, and you stare at two knobs, C and gamma, that nobody ever explained properly.

Turn them the wrong way, and your model memorizes the training set. Turn them the other way, and it draws a boundary so smooth that it ignores the signal. Worse, the two knobs interact, so you cannot tune them one at a time.

This guide explains what each parameter does in plain language, how they combine, and how to tune them with a repeatable recipe. I verified the core intuition against scikit-learn's own RBF SVM example and a quote from the classic LIBSVM practical guide, and I mark which details come from my own knowledge. Numbered brackets like [1] point to the sources at the end. The code is mine and untested, so run it on your data before you trust it.

The Big Picture in One Paragraph

An RBF-kernel SVM separates classes with a flexible, curved boundary. gamma controls how curvy the boundary can get, and C controls how much you punish mistakes on the training data. Both parameters control complexity, from different directions, and the best model balances them.

Parameter	Controls	Too low	Too high
gamma	Reach of one training point	Boundary nearly linear, underfits	Boundary wraps individual points, overfits
C	Price of a training mistake	Wide margin, tolerates errors, underfits	Narrow margin, wiggly boundary, overfits

What gamma Does

The RBF (radial basis function) kernel measures similarity between two points. From my own knowledge, the formula is K(x, x') = exp(-gamma * ||x - x'||^2). Nearby points score close to 1, distant points score close to 0, and gamma sets how fast similarity drops with distance.

Scikit-learn's documentation puts the intuition well: gamma defines how far the influence of a single training example reaches, where low values mean "far" and high values mean "close" [1]. You can also view it as the inverse of the radius of influence of the samples the model selects as support vectors [1].

• Low gamma: each point influences a wide neighbourhood, so the decision boundary stays smooth and simple. Take it too far, and the model underfits.
• High gamma: each point influences only its immediate surroundings, so the boundary wraps tightly around individual points. Take it too far, and the model overfits.

Here is the warning that matters most. According to scikit-learn's example, if gamma is too large, the area of influence of each support vector only includes the support vector itself, and no amount of regularization with C can prevent overfitting [1].

In a very-low-gamma extreme, the same source says the region of influence of any support vector includes the whole training set, so the model behaves much like a linear one [1].

What C Does

C is the penalty for misclassifying training points in the soft-margin SVM. Scikit-learn's example says C behaves as a regularization parameter [1], and my own way of remembering it is that C works like the inverse of regularization strength. That detail surprises people.

• Low C: the model tolerates more training mistakes in exchange for a wider, simpler margin. The boundary stays smoother, which fights overfitting. Too low, and the model underfits.
• High C: the model works hard to classify every training point correctly, even if that means a wiggly boundary and a narrow margin. Too high, and it overfits.

How They Interact

The two parameters do not act independently:

Setting	What happens
High gamma, any C	The kernel itself overfits, and C cannot rescue it [1]
Low gamma, high C	The boundary can still bend, because heavy penalties force a fit within what the smooth kernel allows
Low gamma, low C	Underfits: a boundary close to a straight line and many misclassified points
Moderate gamma, moderate C	Usually the sweet spot, but the exact location depends on your data

The scikit-learn example visualizes this with a grid of decision boundaries and a heatmap of cross-validation scores across C and gamma. It reports that on its example data, the best parameters were C=1.0 and gamma=0.1, with a score of 0.97 [1]. Do not copy those numbers. They belong to that dataset.

A Quick Experiment You Can Run

Build intuition by fitting a few combinations on a toy dataset and comparing training and test scores. This code is mine and untested:

from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=400, noise=0.3, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)

for C in (0.1, 1, 100):
    for gamma in (0.01, 1, 100):
        clf = make_pipeline(StandardScaler(), SVC(C=C, gamma=gamma))
        clf.fit(X_tr, y_tr)
        n_sv = clf.named_steps["svc"].n_support_.sum()
        print(
            f"C={C:<6} gamma={gamma:<6} "
            f"train={clf.score(X_tr, y_tr):.2f} test={clf.score(X_te, y_te):.2f} "
            f"support_vectors={n_sv}"
        )

Expect three patterns:

1. Tiny gamma and tiny C: train and test scores both low, so the model underfits.
2. Huge gamma: train score near perfect, test score noticeably lower, and a large share of points becoming support vectors, so the model overfits.
3. Moderate settings: train and test scores close together and reasonably high.

The support vector count offers a handy diagnostic. A model that needs most of the training set as support vectors is often too complex.

Always Scale Your Features

The RBF kernel depends on distances, so features on large scales dominate. Standardize first, and keep the scaler inside the pipeline so it fits only on training data, as in the code above. Skip this step, and no amount of tuning C and gamma will fix the damage.

From my own knowledge, scikit-learn's default gamma="scale" uses 1 / (n_features * X.var()), which adapts to feature count and variance, while "auto" uses 1 / n_features [3]. Check the current docs for the exact definition in your version.

A Tuning Recipe That Works

The LIBSVM team's practical guide recommends grid search on C and gamma with cross-validation, trying exponentially growing sequences [2]. A forum post quoting the guide gives examples such as C from 2^-5 to 2^15 and gamma from 2^-15 to 2^3 [2]. Scikit-learn's example agrees on the approach: for an initial search, a logarithmic grid with base 10 is often helpful [1].

Here is the recipe:

1. Split off a test set and do not touch it until the end.
2. Build a pipeline with a scaler and SVC.
3. Run a coarse log-scale grid with cross-validation.
4. Check the boundaries. If the best parameters sit on the edge of your grid, extend the grid in that direction [1].
5. Zoom in with a finer grid around the best region.
6. Refit on all training data, and score once on the test set.

import numpy as np
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))

param_grid = {
    "svc__C": np.logspace(-2, 3, 6),
    "svc__gamma": np.logspace(-4, 1, 6),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

search = GridSearchCV(
    pipe, param_grid, cv=cv, scoring="f1_macro",
    n_jobs=-1, return_train_score=True,
)
search.fit(X_train, y_train)

print(search.best_params_, search.best_score_)

# Is the winner on the edge of the grid? If so, extend the grid.
best = search.best_params_
edge = (
    best["svc__C"] in (param_grid["svc__C"][0], param_grid["svc__C"][-1])
    or best["svc__gamma"] in (param_grid["svc__gamma"][0], param_grid["svc__gamma"][-1])
)
print("Best on grid edge:", edge)

The return_train_score=True option lets you compare training and validation scores for every combination. A big gap signals overfitting, and low scores for both signal underfitting.

Tips for a Better Search

• Choose the metric on purpose. Accuracy misleads on imbalanced data. Use F1, balanced accuracy, or area under the curve.
• Use stratified folds for classification.
• Try class_weight="balanced" when classes are skewed.
• Use randomized search when the grid gets large or you add more parameters.
• Consider nested cross-validation for an honest estimate when you compare many models.
• Do not tune on the test set. Ever.

If the grid itself is the bottleneck, the pipeline tuning guide covers randomized and halving searches that reach similar regions with far fewer fits.

Read the Symptoms

Use this cheat sheet when you look at your results:

Symptom	Diagnosis	First fix
High train score, much lower validation score	Overfitting	Lower gamma first, then lower C
Both scores low	Underfitting	Raise gamma and C; check scaling and data quality
Scores flat across a wide region	Data tolerates many settings	Pick a point in the middle of the plateau, not the edge
A sharp peak surrounded by poor scores	Fragile result	Prefer a broader, more stable region
Best value on a grid edge	Optimum outside the grid	Extend the grid [1]
Very many support vectors	Boundary probably too complex	Lower gamma, and check for label noise

A boundary-riding winner is not a good result. It means the search stopped before it found the region, so the reported cross-validation score understates what the model can do and the parameter is not yet identified.

When SVMs Hit Their Limits

Kernel SVMs get expensive as data grows. From my own knowledge, scikit-learn's documentation warns that fit time scales at least quadratically with the number of samples and can become impractical beyond tens of thousands of samples [4]. Check the SVM section of the user guide for the exact wording.

If your dataset is large, consider:

• A linear SVM, which scales much better, if your data separates roughly linearly.
• Kernel approximation methods, like random Fourier features or Nystroem, paired with a linear model.
• Gradient-boosted trees or other models, which often win on tabular data.

Also remember that SVMs do not output calibrated probabilities by default. If you need probabilities, check the calibration options [3].

Mistakes I See Practitioners Make

1. Skipping feature scaling. The kernel then measures nonsense distances.
2. Tuning on a linear scale. Try values like 0.1, 0.2, and 0.3, and you will miss the region that matters. Search on a log scale [1][2].
3. Thinking higher C always means better. It often means overfitting.
4. Ignoring gamma. It usually matters as much as C, or more.
5. Trusting the best cross-validation score on a boundary. Extend the grid.
6. Tuning with the test set. Your score then lies.
7. Using accuracy on imbalanced data. Pick a better metric.
8. Forgetting that fitting scales poorly. Plan for large datasets.
9. Tuning gamma before scaling, then scaling afterwards. The optimal gamma belongs to a specific feature scale, so changing the scaler after tuning invalidates the search.
10. Reusing best_params_ from a different dataset version. The grid optimum moves as the data moves; re-run the search whenever the training set is refreshed.

Your Quick-Start Checklist

1. Scale features inside a pipeline.
2. Hold out a test set.
3. Run a coarse log-scale grid over C and gamma with stratified cross-validation.
4. Extend the grid if the winner sits on an edge.
5. Refine around the best region.
6. Compare train and validation scores, and check the support vector count.
7. Pick a stable region, not a lone peak.
8. Score the test set once.

Wrapping Up

Think of gamma as the reach of each training point, and C as the price of each training mistake. Large values of either one push the model toward memorizing, and small values push it toward oversimplifying. The sweet spot depends on your data, which is why a log-scale grid with cross-validation beats guessing.

Scale your features, search on a log grid, watch the gap between training and validation scores, and extend the grid whenever the winner hugs an edge. Do those four things, and the two mysterious knobs become two ordinary hyperparameters.

If the mechanics of the kernel itself are still fuzzy, the SVM explainer covers margins and support vectors from scratch, and the cross-validation guide is worth reading before you trust any single CV number.

Frequently Asked Questions

What does gamma do in an RBF SVM?

Gamma sets how far the influence of a single training example reaches. The kernel is exp(-gamma * ||x - x'||^2), so a small gamma makes every point influence a wide neighbourhood and the boundary stays smooth, while a large gamma shrinks each point's reach until the boundary wraps around individual points. In scikit-learn's words, low gamma means far and high gamma means close.

What does C do in an SVM?

C is the penalty for misclassifying training points in the soft-margin formulation, and it behaves as a regularization parameter. Low C tolerates more training mistakes in exchange for a wider, simpler margin; high C works hard to classify every point correctly, even at the cost of a wiggly boundary. My own way of remembering it is that C works like the inverse of regularization strength.

Can a very large C stop a large gamma from overfitting?

No. According to scikit-learn's own RBF example, if gamma is too large the area of influence of each support vector only includes the support vector itself, and no amount of regularization with C can prevent overfitting. Gamma sets what the model is capable of expressing; C only chooses how hard it tries within that space.

Why does my SVM perform badly without feature scaling?

Because the RBF kernel is a function of Euclidean distance. A feature measured in thousands dominates one measured in single digits, so the effective gamma differs per feature and the kernel computes nonsense similarities. Standardize inside a pipeline so the scaler sees only the training fold, and no amount of tuning C and gamma will fix a model that skipped this.

Why search C and gamma on a log scale?

Because their effects are multiplicative, not additive. Trying 0.1, 0.2 and 0.3 never reaches the region that matters, while 10^-4, 10^-3 and 10^-2 sweeps orders of magnitude. Both scikit-learn's example and the LIBSVM practical guide recommend exponentially growing sequences, with a base-10 logarithmic grid a common starting point.

What should I do if the best parameters land on the edge of my grid?

Extend the grid in that direction, as scikit-learn's example advises, and search again. A winner on the boundary means the optimum is outside the space you searched, so the reported score understates what the model can do and the parameter is not yet identified. Keep extending until the best point sits comfortably inside the grid.

Why is my train score perfect while validation lags far behind?

That is the overfitting signature, and gamma is the first suspect. High gamma lets the boundary isolate individual points, so training accuracy approaches 100% while test accuracy falls. Lower gamma before you lower C, then check whether the support vector count drops, since a model needing most of the training set as support vectors is usually too complex.

Sources

• [1] scikit-learn, "RBF SVM parameters" — the official example behind the gamma intuition, the interaction warnings, the grid-search advice and the log-scale recommendation: scikit-learn.org/1.6/.../plot_rbf_parameters
• [2] Hsu, Chang and Lin, "A Practical Guide to Support Vector Classification" — the LIBSVM team's recommendation of grid search with cross-validation over exponentially growing sequences, and the 2^-5 to 2^15 ranges: csie.ntu.edu.tw/.../guide.pdf
• [3] scikit-learn, SVC — the gamma values including the "scale" default, probability and class_weight, and the calibration path: scikit-learn.org/.../SVC
• [4] scikit-learn, "Support vector machines" user guide — kernel definitions, scaling guidance, and the complexity note on fit time growing with sample count: scikit-learn.org/modules/svm

Related Articles

• Support Vector Machines Explained: Kernels and Python
• Tuning a Scikit-learn Pipeline Beyond GridSearchCV
• Scikit-learn Cross-Validation: Master K