Your data contains two classes shaped like interleaving crescent moons, so a straight boundary leaves many points on the wrong side. You want a model that can follow the curved separation without treating every training point as equally decisive. A support vector machine gives you a boundary determined by a subset of training examples. Those support vectors make the model's solution easier to inspect.
Only some observations constrain that separation. The moons boundary uses 40 support vectors from 200 training points. We fit linear and kernel SVMs, compare their boundaries with logistic regression, and inspect the retained points to see how the optimisation chooses them.
Quick Win: Linear and Kernel SVM in One Page
Two scikit-learn calls. The first fits a linear SVM to a separable 2D blob; the second fits an RBF SVM to the moons.
import numpy as np
from sklearn.datasets import make_blobs, make_moons
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
# Linearly separable blobs
X_lin, y_lin = make_blobs(n_samples=40, centers=2, cluster_std=0.9,
center_box=(-2.5, 2.5), random_state=1)
y_lin = np.where(y_lin == 0, -1, 1)
linear = SVC(kernel='linear', C=10.0).fit(X_lin, y_lin)
print(f'linear SVM: {len(linear.support_vectors_)} support vectors '
f'out of {len(y_lin)} training points')
# Non-linear moons
X_m, y_m = make_moons(n_samples=200, noise=0.22, random_state=3)
X_m = StandardScaler().fit_transform(X_m)
y_m = np.where(y_m == 0, -1, 1)
rbf = SVC(kernel='rbf', C=4.0, gamma=1.5).fit(X_m, y_m)
print(f'RBF SVM: {len(rbf.support_vectors_)} support vectors '
f'out of {len(y_m)} training points')
Output:
linear SVM: 4 support vectors out of 40 training points
RBF SVM: 40 support vectors out of 200 training points
The linear SVM uses 10% of the training data; the RBF SVM uses 20%. Everything else is irrelevant once training is done. That is what sparse kernel machine means.

This is the classic max-margin picture on real data. Now the kernel version on a problem no straight line can solve:

This is the classic kernel-SVM picture. The curved boundary in the original 2D space comes from a linear hyperplane in a higher-dimensional feature space that the RBF kernel implicitly defines. The next section unpacks how.
What Just Happened: Margin, Lagrangian, Sparsity
Act 1: The maximum margin hyperplane
For a two-class problem with labels , a linear classifier has the form
where is some feature map (identity for the moment). When the classes are linearly separable, infinitely many work. The perceptron picks one based on initialisation and data order. The SVM picks the unique solution that maximises the margin, the perpendicular distance from the boundary to the nearest training point.
After rescaling so the closest points satisfy , the margin equals and the optimisation becomes:
A convex quadratic objective with linear inequality constraints. One global minimum, no local optima to worry about.
Act 2: The Lagrangian dual and where sparsity comes from
Introducing Lagrange multipliers for the margin constraints gives the Lagrangian, then eliminating and produces the dual problem:
maximised subject to and . The kernel is the only place the feature map enters. We never compute explicitly.
The Karush-Kuhn-Tucker (KKT) conditions for this constrained optimisation force, for every training point, either
Points with drop out of the prediction . The remaining points, the support vectors, sit exactly on the margin boundaries. Sparsity is not added as a regulariser; it falls out of the geometry.
You can solve the dual with cvxopt in twenty lines and verify it matches scikit-learn:
import numpy as np
from cvxopt import matrix, solvers
solvers.options['show_progress'] = False
K = X_lin @ X_lin.T # linear kernel
N = len(y_lin)
P = matrix((y_lin[:, None] * y_lin[None, :]) * K)
q = matrix(-np.ones(N))
G = matrix(-np.eye(N))
h = matrix(np.zeros(N))
A = matrix(y_lin.astype(float).reshape(1, -1))
b = matrix(0.0)
sol = solvers.qp(P, q, G, h, A, b)
a = np.ravel(sol['x'])
sv_mask = a > 1e-5
w = ((a * y_lin) @ X_lin)
b_int = np.mean(y_lin[sv_mask] - X_lin[sv_mask] @ w)
print(f'cvxopt: {sv_mask.sum()} SVs, w = {w.round(3)}, b = {b_int:.3f}')
print(f'sklearn: {linear.n_support_.sum()} SVs, '
f'w = {linear.coef_.ravel().round(3)}, b = {linear.intercept_[0]:.3f}')
Output:
cvxopt: 3 SVs, w = [-4.627 -1.843], b = -5.153
sklearn: 4 SVs, w = [-4.626 -1.843], b = -5.151
Same and to four decimal places. The one-SV difference is scikit-learn keeping a borderline point that the hard-margin cvxopt dual zeros out at the chosen tolerance. The dual quadratic program is the SVM, and scikit-learn is just a faster implementation of it (sequential minimal optimisation, Platt 1999) packaged in a friendly API.
Act 3: The kernel trick
Replacing the linear kernel with any positive-definite kernel buys non-linear boundaries without ever materialising the feature map. The classic example is the polynomial kernel , which expands as an inner product in a 6-dimensional feature space. The RBF kernel corresponds to an infinite-dimensional feature space, and we still pay only to evaluate it.

The lift makes the rings linearly separable in 3D. A linear hyperplane there projects back to the original 2D space as a circle: a non-linear boundary, found by a linear algorithm in feature space. The kernel trick is the realisation that we never need itself, only the inner products , which the kernel computes directly from the inputs. Connect this to Gaussian process regression, another method where the kernel matrix does the heavy lifting.
Going Deeper: Soft Margins, C, and Gamma
Real datasets rarely separate cleanly. The hard-margin formulation has no solution when classes overlap, so we introduce slack variables , one per point, and reformulate the objective:
controls the trade-off. Small tolerates slack; large punishes it. In the limit we recover the hard-margin SVM. The dual changes only by adding upper bounds: . Geometrically, controls how wide the margin is allowed to be.

The animated version below sweeps from 0.005 to 200 across twelve frames so you can watch the margin shrink and the support set evolve in real time.

The RBF kernel adds a second knob: , the inverse length-scale. Small produces broad influence regions (smooth boundaries, underfitting); large makes each support vector influence only its immediate neighbourhood (wiggly boundaries, overfitting):

Tuning and together is a 2D grid-search problem. Use GridSearchCV with stratified folds, log-spaced grids ( for both), and the overfitting explorer to build intuition for what overfitting looks like before you start. Always scale your features before fitting an RBF SVM; the kernel is distance-based and a feature with a wider range will dominate .
Hinge vs Cross-Entropy: SVM vs Logistic Regression
The soft-margin SVM can be reformulated as a regularised loss-minimisation problem:
with the hinge loss . Maximum-likelihood logistic regression under target labels minimises an analogous expression with :

The two losses are similar in shape, but the hinge has a flat region for . Confidently correct points contribute zero gradient and zero penalty. Cross-entropy never goes flat, so every point in the dataset pulls the boundary by some non-zero amount. The flat region is the algebraic source of sparsity: the KKT conditions assign to every confidently-correct point, which is exactly the set on which the hinge loss is flat.
When does the choice matter? Side-by-side on overlapping data:

Practical implications:
- Probability outputs. Logistic regression produces calibrated directly. The SVM does not; you have to fit a sigmoid to the decision function as a post-hoc step (Platt 1999), and Tipping (2001) shows the result is poorly calibrated. If you need a probability, reach for logistic regression or a Bayesian neural network instead.
- Sparsity. The SVM tells you which points define the boundary. Strip out the others and prediction is instead of , where is the number of support vectors. On large datasets this matters at inference time.
- Outliers. Both losses upper-bound the 0/1 loss; the squared error from targets does not, and over-penalises confidently-correct points. Avoid squared loss for classification regardless of the algorithm.
Where Sparse Kernel Machines Came From
The SVM has a clean three-paper lineage. Boser, Guyon and Vapnik (1992) introduced the optimal margin classifier with kernels at COLT, building on Vapnik's earlier statistical learning theory work in the USSR (Vapnik & Chervonenkis, 1971). Cortes and Vapnik (1995) added the soft-margin slack variables that made it usable on overlapping data. Platt (1999) gave us sequential minimal optimisation, the algorithm that decomposes the dual QP into pairs of updates and made SVMs tractable on tens of thousands of points; it is still the engine inside libsvm and therefore inside scikit-learn's SVC.
The deeper motivation is structural risk minimisation: among all classifiers that fit the training data, prefer the one with the smallest Vapnik-Chervonenkis dimension. Maximising the margin reduces the VC dimension of the hyperplane class, which bounds the generalisation gap. The geometry (widest gap) and the statistics (tightest generalisation bound) point to the same solution.
The standard textbook references are Vapnik's The Nature of Statistical Learning Theory (1995, Springer), Cristianini and Shawe-Taylor's An Introduction to Support Vector Machines (Cambridge, 2000), and Schölkopf and Smola's Learning with Kernels (MIT, 2002). Burges (1998), "A Tutorial on Support Vector Machines for Pattern Recognition" (Data Mining and Knowledge Discovery, 2(2): 121-167), is the classic 60-page introduction and is freely available online.
A useful contrast with Bayesian model comparison: the SVM picks and by cross-validation, with no marginal-likelihood machinery to weight competing kernel choices automatically. The relevance vector machine (Tipping, 2001) plugs that gap by re-deriving sparsity from a Bayesian prior, recovering posterior probabilities, and typically using fewer basis functions than the SVM.
Limitations and the Bridge to Relevance Vector Machines
Sparse kernel machines are precise and well-grounded, and they have known weak points:
- No native posterior probabilities. The SVM is a decision machine. Platt scaling fits a sigmoid to the decision function as a post-hoc patch, but the calibration is poor, especially in the tails (Tipping, 2001).
- Two-class by construction. Multi-class SVMs are built by combining binary classifiers, either one-versus-rest ( classifiers, with the inconsistency that the underlying values are not on the same scale) or one-versus-one ( classifiers, voted among). Weston and Watkins (1999) propose a single joint objective, but it costs to train.
- Quadratic-to-cubic training cost. The dual QP has variables and full SMO scaling sits between and . SVMs do not scale to the millions-of-rows regime where stochastic gradient logistic regression or neural networks shine.
- Hyperparameter sensitivity. and (or polynomial degree) interact in non-obvious ways and require careful tuning. There is no marginal-likelihood criterion to lean on, just cross-validation.
- Kernel choice is ad hoc. Linear, RBF, polynomial, sigmoid, and string kernels are the standard menu; choosing among them is largely empirical. Kernel composition (, ) extends the menu but does not solve the choice problem.
Use SVMs when: you have moderate-sized data (a few thousand to a few hundred thousand rows), two classes, smooth decision boundaries, no need for calibrated probabilities, and a reason to want a sparse, interpretable support set. Reach for logistic regression when probabilities matter, for tree ensembles when interactions and tabular features dominate, and for relevance vector machines when you need both sparsity and a Bayesian posterior. Use the classification metrics calculator to pick the right operating point on the resulting decision function.
Interactive Tools
- Classification Metrics Calculator: Convert SVM decision-function thresholds into precision, recall, F1, and ROC curves on your own confusion matrix.
- Overfitting Explorer: Build intuition for the bias-variance trade-off that and are quietly arbitrating.
- GP Regression Playground: Adjust the same RBF length-scale in a regression setting to see how kernel width controls smoothness.
Related Posts
- Maximum Likelihood Estimation in Python from Scratch: Logistic regression as MLE under a Bernoulli likelihood, the natural counterpart to the SVM's hinge loss.
- Gaussian Process Regression from Scratch: Another kernel method, where the same machinery powers a non-sparse, fully Bayesian regressor.
- Backprop, Jacobians, Hessians: Why First-Order Wins: Why convex problems like the SVM dual are easier than the non-convex losses of neural networks.
- Bayesian Model Comparison: The Evidence Framework: The marginal-likelihood machinery the SVM lacks, and that the relevance vector machine reintroduces.
- Bayesian Neural Networks: Laplace and the Evidence Framework: The Bayesian recipe for a discriminative model with calibrated uncertainty.
- Tabular Foundation Models: How TabPFN Predicts Without Training: A pre-trained transformer that aims to replace classical tabular classifiers like the SVM.
- SMOTE vs Class Weights: Imbalanced Classification: The hinge-loss margin is sensitive to skewed class ratios; this post covers the resampling and class-weighting fixes that keep the boundary honest.
- Anomaly Detection from Scratch: The one-class SVM turns this margin machinery into a boundary around normal data, flagging whatever falls outside.








