Relevance Vector Machine vs SVM in Python

Build a relevance vector machine from scratch. The RVM gives sparser, calibrated alternatives to the SVM via ARD priors and evidence maximisation.

Cover Image for Relevance Vector Machine vs SVM in Python

The support vector machine from the previous post can separate the two curved classes in our data. Now you want more from the prediction, including a probability and a compact set of points that explains the boundary. A relevance vector machine approaches the same kind of task with a Bayesian model, so we can compare how it selects useful basis functions and expresses uncertainty.

Bayesian fitting can keep that representation smaller. On two-moons classification, the RVM retains ten basis points against the SVM's forty. In regression, it keeps four against the SVR's 29. We build both RVMs in NumPy to trace pruning and the resulting probabilities and uncertainty.

Quick Win: Sparser Than the SVM in Twenty Lines

Open In Colab

The classic sinusoidal regression problem, fit with RVMRegressor from scratch and compared against scikit-learn's SVR.

import numpy as np
from sklearn.svm import SVR
from rvm import RVMRegressor

rng = np.random.default_rng(0)
x = np.linspace(0, 1, 50).reshape(-1, 1)
t = np.sin(2 * np.pi * x.ravel()) + rng.normal(0, 0.1, x.shape[0])

rvm = RVMRegressor(gamma=10.0).fit(x, t)
svr = SVR(kernel='rbf', gamma=10.0, C=10.0, epsilon=0.05).fit(x, t)

print(f'RVM: {rvm.relevance_vector_indices_.size} relevance vectors')
print(f'SVR: {len(svr.support_)} support vectors')

Output:

RVM: 4 relevance vectors
SVR: 29 support vectors

The RVM uses 8% of the training data; the SVR uses 58%. Both fit the underlying sinusoid to comparable test error. That is the headline result of sparse Bayesian learning.

RVM regression on a sinusoid: the red mean curve hugs the true sin(2πx), the pink ribbon shows the ±1σ predictive band, and four blue rings mark the relevance vectors that pin down the entire fit.

The same data, both methods side by side:

Side-by-side: SVR uses 29 support vectors stacked along the sinusoid, while the RVM uses only 4 relevance vectors plus a calibrated ±1σ uncertainty band.

The RVM also delivers the predictive standard deviation σ(x)\sigma(\mathbf{x}) for free. The SVR returns nothing analogous; getting an uncertainty estimate from an SVR requires bootstrap resampling or a separate quantile model. The next section is the whole story of how those four relevance vectors emerged from a starting basis of 51.

Margin, ARD, and the Geometry of Sparsity

Act 1: One precision per weight

The regression form writes the prediction as a linear combination of N+1N+1 basis functions, one bias term plus one RBF kernel centred on each training point:

y(x)=∑n=1Nwnk(x,xn)+b.y(\mathbf{x}) = \sum_{n=1}^{N} w_n k(\mathbf{x}, \mathbf{x}_n) + b.

This looks identical to the SVM dual prediction y(x)=∑nantnk(x,xn)+by(\mathbf{x}) = \sum_n a_n t_n k(\mathbf{x}, \mathbf{x}_n) + b. The difference is the prior on the weights. The SVM has none; sparsity emerges from the hinge loss and the KKT conditions. The RVM puts a zero-mean Gaussian prior on each weight with its own precision hyperparameter αi\alpha_i:

p(w∣α)=∏i=1MN(wi∣0,αi−1).p(\mathbf{w} \mid \boldsymbol{\alpha}) = \prod_{i=1}^{M} \mathcal{N}(w_i \mid 0, \alpha_i^{-1}).

Every weight has its own knob. That is automatic relevance determination: instead of a single shared regulariser strength, the model decides per basis function how much to shrink, and as we will see, "infinitely much" is a valid answer.

Combined with the Gaussian likelihood p(t∣w,β)=∏nN(tn∣w⊤ϕ(xn),β−1)p(\mathbf{t} \mid \mathbf{w}, \beta) = \prod_n \mathcal{N}(t_n \mid \mathbf{w}^\top \boldsymbol{\phi}(\mathbf{x}_n), \beta^{-1}), the posterior over weights is conjugate Gaussian:

p(w∣t,α,β)=N(w∣m,Σ),m=βΣΦ⊤t,Σ=(A+βΦ⊤Φ)−1,p(\mathbf{w} \mid \mathbf{t}, \boldsymbol{\alpha}, \beta) = \mathcal{N}(\mathbf{w} \mid \mathbf{m}, \boldsymbol{\Sigma}), \quad \mathbf{m} = \beta \boldsymbol{\Sigma} \boldsymbol{\Phi}^\top \mathbf{t}, \quad \boldsymbol{\Sigma} = (\mathbf{A} + \beta \boldsymbol{\Phi}^\top \boldsymbol{\Phi})^{-1},

where A=diag(α)\mathbf{A} = \mathrm{diag}(\boldsymbol{\alpha}). So far this is just Bayesian linear regression with a per-weight prior. The RVM trick is in choosing α\boldsymbol{\alpha} and β\beta.

Act 2: Evidence maximisation

We do not put a prior on α\boldsymbol{\alpha} and integrate it out; we set it by maximising the marginal likelihood (the evidence). Integrating out w\mathbf{w} analytically gives the log marginal likelihood:

ln⁡p(t∣X,α,β)=−12{Nln⁡(2π)+ln⁡∣C∣+t⊤C−1t},C=β−1I+ΦA−1Φ⊤.\ln p(\mathbf{t} \mid \mathbf{X}, \boldsymbol{\alpha}, \beta) = -\tfrac{1}{2}\left\{N \ln(2\pi) + \ln |\mathbf{C}| + \mathbf{t}^\top \mathbf{C}^{-1} \mathbf{t}\right\}, \quad \mathbf{C} = \beta^{-1}\mathbf{I} + \boldsymbol{\Phi}\mathbf{A}^{-1}\boldsymbol{\Phi}^\top.

This is the same evidence-framework machinery from Bayesian model comparison and the Bayesian neural network post: score a model by how much probability mass it assigns to the data. Setting derivatives with respect to αi\alpha_i and β\beta to zero gives the elegant re-estimation equations:

αinew=γimi2,(βnew)−1=∥t−Φm∥2N−∑iγi,γi=1−αiΣii.\alpha_i^{\text{new}} = \frac{\gamma_i}{m_i^2}, \qquad (\beta^{\text{new}})^{-1} = \frac{\|\mathbf{t} - \boldsymbol{\Phi}\mathbf{m}\|^2}{N - \sum_i \gamma_i}, \qquad \gamma_i = 1 - \alpha_i \Sigma_{ii}.

The quantity γi\gamma_i measures how well basis function ii is determined by the data: γi≈1\gamma_i \approx 1 means the data pins the weight down precisely, γi≈0\gamma_i \approx 0 means the weight is whatever the prior says it is. The whole RVM fit is a fixed-point iteration of these equations, alternating with the posterior m,Σ\mathbf{m}, \boldsymbol{\Sigma} update.

class RVMRegressor:
    def fit(self, X, t):
        Phi_full = design_matrix(X, X, self.gamma)        # bias column + RBF kernels
        alpha = np.ones(Phi_full.shape[1])
        beta = 1.0 / max(np.var(t) * 0.1, 1e-3)
        active = np.ones_like(alpha, dtype=bool)
        for _ in range(self.max_iter):
            Phi, a = Phi_full[:, active], alpha[active]
            Sigma = np.linalg.inv(np.diag(a) + beta * Phi.T @ Phi)
            m = beta * Sigma @ Phi.T @ t
            gamma_i = 1.0 - a * np.diag(Sigma)
            alpha[active] = np.where(m**2 > 1e-12, gamma_i / m**2, np.inf)
            beta = max(N - gamma_i.sum(), 1e-6) / np.sum((t - Phi @ m) ** 2)
            active = alpha < self.alpha_prune
        # ... store m, Sigma, active for prediction

These are the core fixed-point updates. The notebook also checks convergence, stores fitted parameters and implements prediction.

Act 3: Pruning through evidence maximisation

Run the fit and watch what happens to α\boldsymbol{\alpha}:

Animation showing log10(α) for each basis function over the first 12 iterations. Most grey trajectories climb steadily past the prune threshold of 10⁹, while four red trajectories settle near log10(α) = 0, the four eventual relevance vectors.

Within a few iterations, 47 of the 51 αi\alpha_i have crossed 10910^9 and we prune them. The four survivors stabilise at small αi\alpha_i between 0.2 and 1.3. The pruned weights have posterior p(wi∣t)=N(0,1/αi)p(w_i \mid \mathbf{t}) = \mathcal{N}(0, 1/\alpha_i) with mean and variance both vanishing, so they drop out of the prediction y(x)=∑iwiϕi(x)y(\mathbf{x}) = \sum_i w_i \phi_i(\mathbf{x}) exactly. That is the automatic relevance determination that gives the prior its name.

Why does the marginal likelihood prefer to send αi\alpha_i to infinity? A small two-point example makes the geometry concrete. Consider a two-point dataset, one basis function φ=(φ(x1),φ(x2))⊤\boldsymbol{\varphi} = (\varphi(\mathbf{x}_1), \varphi(\mathbf{x}_2))^\top, and the marginal-likelihood covariance C=β−1I+α−1φφ⊤\mathbf{C} = \beta^{-1}\mathbf{I} + \alpha^{-1}\boldsymbol{\varphi}\boldsymbol{\varphi}^\top:

Two-panel sketch: the left panel shows α = ∞, where C is just the isotropic noise circle β⁻¹I and the target t sits at distance √4.5 from the origin. The right panel shows finite α: C is stretched along φ but in this misalignment the density at t is unchanged or lower. The marginal likelihood prefers the panel with higher density at t.

When the basis vector φ\boldsymbol{\varphi} aligns with the target vector t\mathbf{t}, finite α\alpha stretches C\mathbf{C} toward t\mathbf{t} and the density p(t∣α,β)p(\mathbf{t} \mid \alpha, \beta) rises. When the basis vector is poorly aligned, finite α\alpha stretches C\mathbf{C} away from t\mathbf{t}, and the density drops below the isotropic noise model. In that case the marginal likelihood is maximised at α→∞\alpha \to \infty, the basis function gets pruned, and only well-aligned bases survive. Sparsity is not an extra penalty term; it is what evidence maximisation does on its own.

Sequential Algorithm, Classification, and Limits

From iteration to closed form

The naive iteration above couples every αi\alpha_i through Σ\boldsymbol{\Sigma}, so even a single update requires inverting the full M×MM \times M matrix. Faul and Tipping (2002) noticed that the dependence of the marginal likelihood on a single αi\alpha_i can be made explicit. Pull the contribution of basis function ii out of C\mathbf{C}:

C=C−i+αi−1φiφi⊤,dλ(αi)dαi=αi−1si2−(qi2−si)2(αi+si)2,\mathbf{C} = \mathbf{C}_{-i} + \alpha_i^{-1}\boldsymbol{\varphi}_i\boldsymbol{\varphi}_i^\top, \quad \frac{\mathrm{d}\lambda(\alpha_i)}{\mathrm{d}\alpha_i} = \frac{\alpha_i^{-1} s_i^2 - (q_i^2 - s_i)}{2(\alpha_i + s_i)^2},

with si=φi⊤C−i−1φis_i = \boldsymbol{\varphi}_i^\top \mathbf{C}_{-i}^{-1} \boldsymbol{\varphi}_i (the sparsity factor) and qi=φi⊤C−i−1tq_i = \boldsymbol{\varphi}_i^\top \mathbf{C}_{-i}^{-1} \mathbf{t} (the quality factor). Setting the derivative to zero gives a closed form:

αi=si2qi2−siif qi2>si,αi=∞otherwise.\alpha_i = \frac{s_i^2}{q_i^2 - s_i} \quad \text{if } q_i^2 > s_i, \qquad \alpha_i = \infty \quad \text{otherwise}.

A basis function deserves a place in the model precisely when its alignment with the residual (the quality) exceeds its overlap with the bases already in the model (the sparsity). The corresponding fast marginal likelihood maximisation algorithm (Tipping and Faul, 2003) considers basis functions one at a time, adding them when q2>sq^2 > s and pruning them otherwise. Each step costs O(M3)O(M^3) where MM is the active basis count, typically a few dozen; a full RVM fit with N=1000N = 1000 training points is dominated by a handful of these cheap updates rather than a single inversion of a 1000×10001000 \times 1000 matrix.

Classification: Laplace plus IRLS

The classification form replaces the linear output with a logistic sigmoid:

y(x,w)=σ ⁣(w⊤ϕ(x)).y(\mathbf{x}, \mathbf{w}) = \sigma\!\left(\mathbf{w}^\top \boldsymbol{\phi}(\mathbf{x})\right).

The likelihood is no longer Gaussian, so we cannot integrate out w\mathbf{w} analytically. Instead we use a Laplace approximation around the posterior mode (the same trick from the Bayesian neural network post), found by iterative reweighted least squares:

def _irls(self, Phi, t, alpha):
    w = np.zeros(Phi.shape[1])
    A = np.diag(alpha)
    for _ in range(self.irls_iter):
        y = sigmoid(Phi @ w)
        B = y * (1.0 - y)
        g = Phi.T @ (t - y) - alpha * w
        H = -(Phi.T @ (B[:, None] * Phi) + A)
        w = w - np.linalg.solve(H, g)
    Sigma = np.linalg.inv(Phi.T @ (B[:, None] * Phi) + A)
    return w, Sigma

Then the same αinew=γi/(wi⋆)2\alpha_i^{\text{new}} = \gamma_i / (w_i^\star)^2 update from the regression case applies. Run this on the moons dataset and compare to the SVM:

Side-by-side classification on two moons. Left: SVM with C=4 and γ=1.5 selects 40 support vectors clustered along the boundary. Right: RVM with γ=1.0 selects only 10 relevance vectors, sitting away from the boundary in regions of high prior alignment.

The RVM relevance vectors do not cluster near the decision boundary the way SVM support vectors do. The reason: a basis function centred near the boundary has φn\boldsymbol{\varphi}_n poorly aligned with the target vector t\mathbf{t} (it predicts class membership for points on either side equally), so it tends to be pruned. The RVM picks prototypes for each region of the input space rather than guards along the dividing line.

The Laplace approximation also gives us a calibrated posterior p(t=1∣x)p(t = 1 \mid \mathbf{x}) directly, with the MacKay (1992) approximation σ ⁣(κ(x) m⊤ϕ(x))\sigma\!\left(\kappa(\mathbf{x}) \, \mathbf{m}^\top \boldsymbol{\phi}(\mathbf{x})\right) where κ=(1+πσ2/8)−1/2\kappa = (1 + \pi \sigma^2 / 8)^{-1/2} shrinks confidence in regions of high posterior variance:

RVM posterior probability heatmap: the red and blue colours show p(t=1|x) ranging from 0 to 1, and dashed contours mark the 10%, 25%, 75%, 90% confidence levels. Probabilities transition smoothly across the boundary and decay to roughly 0.5 in the corners far from any training data.

The probabilities are properly calibrated near the data and shrink toward 0.5 in the corners far from any training point. Compare this to Platt-scaled SVM probabilities, which Tipping (2001) shows are systematically over-confident in those same far-from-data regions.

Sparsity is consistent across noise levels

The sparsity gap stays roughly constant as the dataset gets noisier:

Two-panel sweep across noise levels 0.10 to 0.45 on the moons dataset. Left: SVM support-vector counts grow from 19 to 88, RVM relevance-vector counts stay between 10 and 15. Right: training accuracies stay within one percentage point of each other across all noise levels.

The SVM hard-margin set explodes with noise because every misclassified point ends up inside the margin and contributes a positive ana_n. The RVM keeps a small fixed set of prototype points, and the marginal-likelihood criterion does not care that some of them are misclassified: what matters is the alignment of φn\boldsymbol{\varphi}_n with the target vector, not the location of xn\mathbf{x}_n relative to a margin.

Where Sparse Bayesian Learning Came From

The RVM is the practical face of sparse Bayesian learning, a research thread that started with MacKay's evidence framework in the early 1990s. MacKay (1992) introduced the type-2 maximum-likelihood approach to setting hyperparameters in Bayesian regression, the same mechanism now used everywhere from Gaussian process kernel learning to LASSO-style ARD priors. The connection to MacKay's PhD thesis is explicit: Tipping was MacKay's PhD student.

The seminal paper, Tipping (2001), "Sparse Bayesian Learning and the Relevance Vector Machine", Journal of Machine Learning Research 1, 211-244, is freely available online and is the right place to start. It introduces the model, derives the regression and classification cases, and demonstrates the sparsity-versus-SVM result on benchmark datasets. Faul and Tipping (2002), "Analysis of Sparse Bayesian Learning", in NIPS 14, supplies the closed-form q2>sq^2 > s analysis that turned the RVM from a slow type-2 ML iteration into a usable algorithm, and Tipping and Faul (2003), "Fast Marginal Likelihood Maximisation for Sparse Bayesian Models", AI & Statistics 2003, gives the sequential algorithm that is the practical implementation.

Two natural neighbours sit alongside the RVM. Gaussian process regression is the non-sparse Bayesian counterpart: GPs have the same kernel matrix Kij=k(xi,xj)K_{ij} = k(\mathbf{x}_i, \mathbf{x}_j) and the same evidence framework for hyperparameter learning, but they keep all NN training points and integrate over the function values directly, paying O(N3)O(N^3) for the inversion. The RVM throws away most of the kernel matrix at the cost of breaking the proper marginal likelihood (the ARD prior is fit by point estimation, not integration, so it is technically not a Gaussian process). The relationship between the two is laid out by Rasmussen and Quiñonero-Candela (2005), "A Unifying View of Sparse Approximate Gaussian Process Regression", in JMLR 6, 1939-1959.

The Bayesian counterpart on the discrete side is the naive Bayes / TF-IDF classifier where the kernel collapses to a coordinate basis and ARD is replaced by Bayesian feature selection. The shared message is that putting a prior on the complexity of the model, rather than on the model's outputs, is what produces interpretable sparsity. The SVM gets its sparsity from the hinge loss; the RVM gets it from the prior; the lasso gets it from the L1 penalty. They all end up zeroing out coefficients, but only the RVM gives you a calibrated posterior alongside the sparse coefficient vector.

Limitations and Trade-offs Against the SVM

The RVM is not a free lunch. Five practical caveats are worth taking seriously:

  • Non-convex training. The marginal-likelihood objective is not concave in α\boldsymbol{\alpha}. Different initialisations can converge to different sparse subsets, especially on datasets where multiple basis functions are nearly redundant. The fast Tipping-Faul algorithm partly mitigates this by starting from a single well-aligned basis, but the global optimum is not guaranteed.
  • Slower training than the SVM. Even with the fast algorithm, RVM training is roughly two to ten times slower than an SVM on comparable data, because each iteration computes a posterior covariance rather than solving a coordinate-descent QP. Tipping (2001) reports the trade-off as "longer training, much faster prediction" thanks to the much smaller relevance vector set.
  • Overconfident extrapolation. With basis functions centred only on training points, predictive variance shrinks in input regions far from any training point, which is the opposite of what Gaussian process regression does. An RVM far outside the training domain can be confidently wrong. Use the GP regression playground to feel the contrast directly.
  • Multi-class scaling. The two-class Laplace approximation generalises to KK classes with a MK×MKMK \times MK Hessian, giving an extra K3K^3 factor in training cost. For more than a handful of classes, the SVM's one-versus-one or one-versus-rest ensembles are practically faster, even if less principled.
  • No off-the-shelf scikit-learn implementation. Unlike sklearn.svm.SVC, there is no RelevanceVectorClassifier in mainstream Python ecosystems. Third-party packages (sklearn-rvm, skrvm) exist but are unmaintained. The accompanying notebook includes a NumPy implementation.

When to reach for an RVM: medium-sized datasets (thousands of points, not millions), two-class or low-KK multi-class problems, and use cases that need calibrated probabilities or extreme prediction-time sparsity. When to stick with the SVM: very large datasets where SMO's O(N2)O(N^2) scales better than the RVM's per-iteration matrix inversion, or when a non-Bayesian black-box decision function is enough. When to skip both for a Gaussian process: small-to-medium data with a need for properly Bayesian extrapolation away from the training domain.

Interactive Tools

  • Bayes' Theorem Calculator: Build intuition for the marginal-likelihood machinery the RVM uses to drive αi→∞\alpha_i \to \infty.
  • GP Regression Playground: Compare the same RBF kernel in a fully Bayesian, non-sparse setting and feel the difference in extrapolation behaviour.
  • Overfitting Explorer: See bias-variance trade-offs in action; the RVM's evidence framework arbitrates this trade-off automatically.

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions