The support vector machine from the previous post can separate the two curved classes in our data. Now you want more from the prediction, including a probability and a compact set of points that explains the boundary. A relevance vector machine approaches the same kind of task with a Bayesian model, so we can compare how it selects useful basis functions and expresses uncertainty.
Bayesian fitting can keep that representation smaller. On two-moons classification, the RVM retains ten basis points against the SVM's forty. In regression, it keeps four against the SVR's 29. We build both RVMs in NumPy to trace pruning and the resulting probabilities and uncertainty.
Quick Win: Sparser Than the SVM in Twenty Lines
The classic sinusoidal regression problem, fit with RVMRegressor from scratch and compared against scikit-learn's SVR.
import numpy as np
from sklearn.svm import SVR
from rvm import RVMRegressor
rng = np.random.default_rng(0)
x = np.linspace(0, 1, 50).reshape(-1, 1)
t = np.sin(2 * np.pi * x.ravel()) + rng.normal(0, 0.1, x.shape[0])
rvm = RVMRegressor(gamma=10.0).fit(x, t)
svr = SVR(kernel='rbf', gamma=10.0, C=10.0, epsilon=0.05).fit(x, t)
print(f'RVM: {rvm.relevance_vector_indices_.size} relevance vectors')
print(f'SVR: {len(svr.support_)} support vectors')
Output:
RVM: 4 relevance vectors
SVR: 29 support vectors
The RVM uses 8% of the training data; the SVR uses 58%. Both fit the underlying sinusoid to comparable test error. That is the headline result of sparse Bayesian learning.

The same data, both methods side by side:

The RVM also delivers the predictive standard deviation for free. The SVR returns nothing analogous; getting an uncertainty estimate from an SVR requires bootstrap resampling or a separate quantile model. The next section is the whole story of how those four relevance vectors emerged from a starting basis of 51.
Margin, ARD, and the Geometry of Sparsity
Act 1: One precision per weight
The regression form writes the prediction as a linear combination of basis functions, one bias term plus one RBF kernel centred on each training point:
This looks identical to the SVM dual prediction . The difference is the prior on the weights. The SVM has none; sparsity emerges from the hinge loss and the KKT conditions. The RVM puts a zero-mean Gaussian prior on each weight with its own precision hyperparameter :
Every weight has its own knob. That is automatic relevance determination: instead of a single shared regulariser strength, the model decides per basis function how much to shrink, and as we will see, "infinitely much" is a valid answer.
Combined with the Gaussian likelihood , the posterior over weights is conjugate Gaussian:
where . So far this is just Bayesian linear regression with a per-weight prior. The RVM trick is in choosing and .
Act 2: Evidence maximisation
We do not put a prior on and integrate it out; we set it by maximising the marginal likelihood (the evidence). Integrating out analytically gives the log marginal likelihood:
This is the same evidence-framework machinery from Bayesian model comparison and the Bayesian neural network post: score a model by how much probability mass it assigns to the data. Setting derivatives with respect to and to zero gives the elegant re-estimation equations:
The quantity measures how well basis function is determined by the data: means the data pins the weight down precisely, means the weight is whatever the prior says it is. The whole RVM fit is a fixed-point iteration of these equations, alternating with the posterior update.
class RVMRegressor:
def fit(self, X, t):
Phi_full = design_matrix(X, X, self.gamma) # bias column + RBF kernels
alpha = np.ones(Phi_full.shape[1])
beta = 1.0 / max(np.var(t) * 0.1, 1e-3)
active = np.ones_like(alpha, dtype=bool)
for _ in range(self.max_iter):
Phi, a = Phi_full[:, active], alpha[active]
Sigma = np.linalg.inv(np.diag(a) + beta * Phi.T @ Phi)
m = beta * Sigma @ Phi.T @ t
gamma_i = 1.0 - a * np.diag(Sigma)
alpha[active] = np.where(m**2 > 1e-12, gamma_i / m**2, np.inf)
beta = max(N - gamma_i.sum(), 1e-6) / np.sum((t - Phi @ m) ** 2)
active = alpha < self.alpha_prune
# ... store m, Sigma, active for prediction
These are the core fixed-point updates. The notebook also checks convergence, stores fitted parameters and implements prediction.
Act 3: Pruning through evidence maximisation
Run the fit and watch what happens to :

Within a few iterations, 47 of the 51 have crossed and we prune them. The four survivors stabilise at small between 0.2 and 1.3. The pruned weights have posterior with mean and variance both vanishing, so they drop out of the prediction exactly. That is the automatic relevance determination that gives the prior its name.
Why does the marginal likelihood prefer to send to infinity? A small two-point example makes the geometry concrete. Consider a two-point dataset, one basis function , and the marginal-likelihood covariance :

When the basis vector aligns with the target vector , finite stretches toward and the density rises. When the basis vector is poorly aligned, finite stretches away from , and the density drops below the isotropic noise model. In that case the marginal likelihood is maximised at , the basis function gets pruned, and only well-aligned bases survive. Sparsity is not an extra penalty term; it is what evidence maximisation does on its own.
Sequential Algorithm, Classification, and Limits
From iteration to closed form
The naive iteration above couples every through , so even a single update requires inverting the full matrix. Faul and Tipping (2002) noticed that the dependence of the marginal likelihood on a single can be made explicit. Pull the contribution of basis function out of :
with (the sparsity factor) and (the quality factor). Setting the derivative to zero gives a closed form:
A basis function deserves a place in the model precisely when its alignment with the residual (the quality) exceeds its overlap with the bases already in the model (the sparsity). The corresponding fast marginal likelihood maximisation algorithm (Tipping and Faul, 2003) considers basis functions one at a time, adding them when and pruning them otherwise. Each step costs where is the active basis count, typically a few dozen; a full RVM fit with training points is dominated by a handful of these cheap updates rather than a single inversion of a matrix.
Classification: Laplace plus IRLS
The classification form replaces the linear output with a logistic sigmoid:
The likelihood is no longer Gaussian, so we cannot integrate out analytically. Instead we use a Laplace approximation around the posterior mode (the same trick from the Bayesian neural network post), found by iterative reweighted least squares:
def _irls(self, Phi, t, alpha):
w = np.zeros(Phi.shape[1])
A = np.diag(alpha)
for _ in range(self.irls_iter):
y = sigmoid(Phi @ w)
B = y * (1.0 - y)
g = Phi.T @ (t - y) - alpha * w
H = -(Phi.T @ (B[:, None] * Phi) + A)
w = w - np.linalg.solve(H, g)
Sigma = np.linalg.inv(Phi.T @ (B[:, None] * Phi) + A)
return w, Sigma
Then the same update from the regression case applies. Run this on the moons dataset and compare to the SVM:

The RVM relevance vectors do not cluster near the decision boundary the way SVM support vectors do. The reason: a basis function centred near the boundary has poorly aligned with the target vector (it predicts class membership for points on either side equally), so it tends to be pruned. The RVM picks prototypes for each region of the input space rather than guards along the dividing line.
The Laplace approximation also gives us a calibrated posterior directly, with the MacKay (1992) approximation where shrinks confidence in regions of high posterior variance:

The probabilities are properly calibrated near the data and shrink toward 0.5 in the corners far from any training point. Compare this to Platt-scaled SVM probabilities, which Tipping (2001) shows are systematically over-confident in those same far-from-data regions.
Sparsity is consistent across noise levels
The sparsity gap stays roughly constant as the dataset gets noisier:

The SVM hard-margin set explodes with noise because every misclassified point ends up inside the margin and contributes a positive . The RVM keeps a small fixed set of prototype points, and the marginal-likelihood criterion does not care that some of them are misclassified: what matters is the alignment of with the target vector, not the location of relative to a margin.
Where Sparse Bayesian Learning Came From
The RVM is the practical face of sparse Bayesian learning, a research thread that started with MacKay's evidence framework in the early 1990s. MacKay (1992) introduced the type-2 maximum-likelihood approach to setting hyperparameters in Bayesian regression, the same mechanism now used everywhere from Gaussian process kernel learning to LASSO-style ARD priors. The connection to MacKay's PhD thesis is explicit: Tipping was MacKay's PhD student.
The seminal paper, Tipping (2001), "Sparse Bayesian Learning and the Relevance Vector Machine", Journal of Machine Learning Research 1, 211-244, is freely available online and is the right place to start. It introduces the model, derives the regression and classification cases, and demonstrates the sparsity-versus-SVM result on benchmark datasets. Faul and Tipping (2002), "Analysis of Sparse Bayesian Learning", in NIPS 14, supplies the closed-form analysis that turned the RVM from a slow type-2 ML iteration into a usable algorithm, and Tipping and Faul (2003), "Fast Marginal Likelihood Maximisation for Sparse Bayesian Models", AI & Statistics 2003, gives the sequential algorithm that is the practical implementation.
Two natural neighbours sit alongside the RVM. Gaussian process regression is the non-sparse Bayesian counterpart: GPs have the same kernel matrix and the same evidence framework for hyperparameter learning, but they keep all training points and integrate over the function values directly, paying for the inversion. The RVM throws away most of the kernel matrix at the cost of breaking the proper marginal likelihood (the ARD prior is fit by point estimation, not integration, so it is technically not a Gaussian process). The relationship between the two is laid out by Rasmussen and Quiñonero-Candela (2005), "A Unifying View of Sparse Approximate Gaussian Process Regression", in JMLR 6, 1939-1959.
The Bayesian counterpart on the discrete side is the naive Bayes / TF-IDF classifier where the kernel collapses to a coordinate basis and ARD is replaced by Bayesian feature selection. The shared message is that putting a prior on the complexity of the model, rather than on the model's outputs, is what produces interpretable sparsity. The SVM gets its sparsity from the hinge loss; the RVM gets it from the prior; the lasso gets it from the L1 penalty. They all end up zeroing out coefficients, but only the RVM gives you a calibrated posterior alongside the sparse coefficient vector.
Limitations and Trade-offs Against the SVM
The RVM is not a free lunch. Five practical caveats are worth taking seriously:
- Non-convex training. The marginal-likelihood objective is not concave in . Different initialisations can converge to different sparse subsets, especially on datasets where multiple basis functions are nearly redundant. The fast Tipping-Faul algorithm partly mitigates this by starting from a single well-aligned basis, but the global optimum is not guaranteed.
- Slower training than the SVM. Even with the fast algorithm, RVM training is roughly two to ten times slower than an SVM on comparable data, because each iteration computes a posterior covariance rather than solving a coordinate-descent QP. Tipping (2001) reports the trade-off as "longer training, much faster prediction" thanks to the much smaller relevance vector set.
- Overconfident extrapolation. With basis functions centred only on training points, predictive variance shrinks in input regions far from any training point, which is the opposite of what Gaussian process regression does. An RVM far outside the training domain can be confidently wrong. Use the GP regression playground to feel the contrast directly.
- Multi-class scaling. The two-class Laplace approximation generalises to classes with a Hessian, giving an extra factor in training cost. For more than a handful of classes, the SVM's one-versus-one or one-versus-rest ensembles are practically faster, even if less principled.
- No off-the-shelf scikit-learn implementation. Unlike
sklearn.svm.SVC, there is noRelevanceVectorClassifierin mainstream Python ecosystems. Third-party packages (sklearn-rvm,skrvm) exist but are unmaintained. The accompanying notebook includes a NumPy implementation.
When to reach for an RVM: medium-sized datasets (thousands of points, not millions), two-class or low- multi-class problems, and use cases that need calibrated probabilities or extreme prediction-time sparsity. When to stick with the SVM: very large datasets where SMO's scales better than the RVM's per-iteration matrix inversion, or when a non-Bayesian black-box decision function is enough. When to skip both for a Gaussian process: small-to-medium data with a need for properly Bayesian extrapolation away from the training domain.
Interactive Tools
- Bayes' Theorem Calculator: Build intuition for the marginal-likelihood machinery the RVM uses to drive .
- GP Regression Playground: Compare the same RBF kernel in a fully Bayesian, non-sparse setting and feel the difference in extrapolation behaviour.
- Overfitting Explorer: See bias-variance trade-offs in action; the RVM's evidence framework arbitrates this trade-off automatically.
Related Posts
- Support Vector Machines: A Sparse Kernel Machine Guide: The SVM whose limitations the RVM is designed to fix.
- Bayesian Model Comparison: The Evidence Framework: The evidence framework that the RVM uses to choose and automatically.
- Bayesian Neural Networks: Laplace and the Evidence Framework: The same Laplace approximation we use here for RVM classification, applied to neural networks.
- Gaussian Process Regression from Scratch: The non-sparse Bayesian kernel cousin of the RVM, with no pruning but proper extrapolation uncertainty.
- Maximum Likelihood Estimation in Python from Scratch: The ML grounding from which the type-2 ML / evidence framework descends.





