Gaussian Process Regression: The Bayesian Approach to Curve Fitting

Gaussian process regressor from scratch in NumPy. How kernels encode smoothness, posteriors collapse with data, and GPs power Bayesian optimisation.

Cover Image for Gaussian Process Regression: The Bayesian Approach to Curve Fitting

You're comparing model settings, and each training run takes long enough that you cannot try every possibility. A handful of completed runs gives you some evidence, but leaves large gaps between the settings you tested. Gaussian process regression offers a way to estimate values in those gaps while also describing uncertainty. This makes it useful when the next measurement or experiment needs to be chosen carefully.

We fit a curve to turn gaps between observations into a prediction and variance. At input 0.2, the GP predicts about 0.98 with variance 0.21. Kernel conditioning links observations to that estimate, following the model used by our hyperparameter optimisation example.

This is regression with built-in uncertainty quantification. Unlike fitting a line or a polynomial, a GP doesn't commit to a fixed functional form. Instead, it defines a distribution over functions and lets the data narrow it down. The result is a smooth prediction with a confidence band that widens where data is sparse and tightens where it's dense.

By the end of this post, you'll implement GP regression from scratch with NumPy, understand how the kernel function encodes your assumptions about smoothness, and see exactly why GPs power Bayesian optimisation.

Let's Build It

Click the badge to open the interactive notebook:

Open In Colab

Watch how the GP posterior sharpens as we feed it observations one by one. The shaded region represents 95% confidence, and it collapses around each data point:

GP posterior building up as observations are added one by one. The uncertainty band starts wide (prior) and collapses around each new data point.

Here's the complete implementation. We'll use 6 noisy observations from Ebden's GP tutorial (2008) and predict across a dense grid:

import numpy as np
from numpy.linalg import inv, det
import matplotlib.pyplot as plt

# --- The Kernel ---
def rbf_kernel(x1, x2, sigma_f, l):
    """Squared exponential (RBF) kernel."""
    return sigma_f**2 * np.exp(-0.5 * (x1 - x2)**2 / l**2)

def kernel_matrix(X1, X2, sigma_f, l):
    """Compute kernel matrix between two sets of points."""
    return np.array([[rbf_kernel(a, b, sigma_f, l) for b in X2] for a in X1])

# --- GP Prediction (Ebden Eq. 8 & 9) ---
def gp_predict(X_train, y_train, X_test, sigma_f, l, sigma_n):
    """Predict mean and variance at test points."""
    n = len(X_train)
    K = kernel_matrix(X_train, X_train, sigma_f, l) + sigma_n**2 * np.eye(n)
    K_s = kernel_matrix(X_test, X_train, sigma_f, l)
    K_ss = kernel_matrix(X_test, X_test, sigma_f, l) + sigma_n**2 * np.eye(len(X_test))

    K_inv = inv(K)
    mu = K_s @ K_inv @ y_train
    cov = K_ss - K_s @ K_inv @ K_s.T

    return mu, cov

# --- Data from Ebden (2008) tutorial ---
X_train = np.array([-1.50, -1.00, -0.75, -0.40, -0.25, 0.00])
y_train = 0.55 * np.array([-3.0, -2.0, -0.6, 0.4, 1.0, 1.6])

# Hyperparameters (optimised values from the tutorial: sigma_f=1.27, l=1.0)
sigma_n = 0.3   # observation noise (known from error bars)
sigma_f = 1.27   # signal standard deviation (optimised)
l = 1.0          # length-scale (optimised)

# Predict on a dense grid
X_test = np.linspace(-2.0, 1.0, 200)
mu, cov = gp_predict(X_train, y_train, X_test, sigma_f, l, sigma_n)
std = np.sqrt(np.diag(cov))

# Plot
fig, ax = plt.subplots(figsize=(10, 5))
ax.fill_between(X_test, mu - 1.96 * std, mu + 1.96 * std,
                alpha=0.2, color='tab:blue', label='95% confidence')
ax.plot(X_test, mu, 'tab:blue', linewidth=2, label='GP mean')
ax.errorbar(X_train, y_train, yerr=sigma_n, fmt='ro', capsize=4,
            markersize=6, label='Training data')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend()
plt.show()

GP posterior: smooth mean prediction (blue line) with 95% confidence band (shaded) passing through 6 noisy observations (red dots with error bars).

The blue line is our best estimate. The shaded band is the 95% confidence interval: wide where we have no data, narrow near the observations. At x∗=0.2x_* = 0.2, the GP predicts yˉ∗≈0.98\bar{y}_* \approx 0.98 with variance 0.21, closely matching Ebden's worked example (yˉ∗=0.95\bar{y}_* = 0.95, var=0.21\text{var} = 0.21).

What Just Happened?

Three ingredients make GP regression work: a kernel function that encodes our smoothness assumptions, covariance matrices that capture relationships between all points, and conditioning that turns the prior into a posterior.

The Kernel: Encoding Smoothness

The squared exponential (RBF) kernel measures similarity between inputs:

k(x,x′)=σf2exp⁡ ⁣(−(x−x′)22ℓ2)k(x, x') = \sigma_f^2 \exp\!\left(-\frac{(x - x')^2}{2\ell^2}\right)

Think of it as answering: "If I know the function value at xx, how much does that tell me about x′x'?"

  • Close together (∣x−x′∣≪ℓ|x - x'| \ll \ell): k≈σf2k \approx \sigma_f^2 (strong correlation, they "see" each other)
  • Far apart (∣x−x′∣≫ℓ|x - x'| \gg \ell): k≈0k \approx 0 (independent, no information shared)

The two hyperparameters control different things:

  • σf\sigma_f (signal std): the typical amplitude of the function. Higher means larger vertical swings.
  • ℓ\ell (length-scale): how far you need to move in xx before the function value changes significantly. Short ℓ\ell gives wiggly functions; long ℓ\ell gives smooth ones.

There's also σn\sigma_n (noise std), capturing measurement noise: each observation yy relates to the true function via y=f(x)+N(0,σn2)y = f(x) + \mathcal{N}(0, \sigma_n^2).

The effect of the length-scale is dramatic:

Three GP regression fits with different length-scales: short (l=0.3, wiggly), medium (l=1.0, smooth), and long (l=3.0, over-smoothed).

With ℓ=0.3\ell = 0.3, the GP tries to wiggle through every point; it over-fits. With ℓ=3.0\ell = 3.0, it's so smooth it can't follow the data's trend; it under-fits. The optimised ℓ=1.0\ell = 1.0 strikes the right balance.

Building the Covariance Matrices

To make predictions, we compute three kernel matrices:

KK, between all training points (Ebden Eq. 4):

Kij=k(xi,xj)+σn2δijK_{ij} = k(x_i, x_j) + \sigma_n^2 \delta_{ij}

The σn2\sigma_n^2 on the diagonal accounts for observation noise. Off-diagonal entries capture how correlated two training points are.

K∗K_*, between test and training points (Ebden Eq. 5):

K∗=[k(x∗,x1)k(x∗,x2)⋯k(x∗,xn)]K_* = \begin{bmatrix} k(x_*, x_1) & k(x_*, x_2) & \cdots & k(x_*, x_n) \end{bmatrix}

K∗∗K_{**}, between test points:

K∗∗=k(x∗,x∗)K_{**} = k(x_*, x_*)

In our code, kernel_matrix computes these. The + sigma_n**2 * np.eye(n) adds noise to the diagonal.

The Prediction Equations

The GP assumption is that training outputs y\mathbf{y} and test outputs y∗y_* are jointly Gaussian:

[yy∗]∼N ⁣(0,[KK∗⊤K∗K∗∗])\begin{bmatrix} \mathbf{y} \\ y_* \end{bmatrix} \sim \mathcal{N}\!\left(\mathbf{0}, \begin{bmatrix} K & K_*^\top \\ K_* & K_{**} \end{bmatrix}\right)

Conditioning on the observed y\mathbf{y} gives the posterior, also Gaussian (this is the beauty of Gaussians: conditioning preserves Gaussianity):

yˉ∗=K∗K−1y\bar{y}_* = K_* K^{-1} \mathbf{y}
var(y∗)=K∗∗−K∗K−1K∗⊤\text{var}(y_*) = K_{**} - K_* K^{-1} K_*^\top

The mean yˉ∗\bar{y}_* is a weighted combination of the training outputs, where the weights come from how correlated x∗x_* is with each training point. The variance shrinks where K∗K_* has large entries (near training data) and expands where x∗x_* is far from any observation.

This is Bayesian inference in action: the prior (encoded by the kernel) gets updated by the data to produce a posterior that's both a prediction and an honest assessment of uncertainty.

Going Deeper

GP Prior: What the Kernel Believes Before Seeing Data

Before any observations, the GP prior defines a distribution over functions. We can sample from it by drawing from N(0,K)\mathcal{N}(\mathbf{0}, K) where KK is the kernel matrix evaluated on a grid:

X_grid = np.linspace(-2, 2, 200)
K_prior = kernel_matrix(X_grid, X_grid, sigma_f=1.27, l=1.0)
K_prior += 1e-8 * np.eye(len(X_grid))  # numerical stability

# Draw 3 random functions from the prior
np.random.seed(42)
samples = np.random.multivariate_normal(np.zeros(len(X_grid)), K_prior, size=5)

fig, ax = plt.subplots(figsize=(10, 5))
for i in range(5):
    ax.plot(X_grid, samples[i], label=f'Sample {i+1}')
ax.fill_between(X_grid, -1.96 * 1.27, 1.96 * 1.27, alpha=0.1, color='grey',
                label='95% prior band')
ax.set_xlabel('x')
ax.set_ylabel('f(x)')
ax.legend()
plt.show()

Three random functions sampled from the GP prior. All are smooth (determined by the length-scale) with amplitudes controlled by sigma_f.

Every sample is smooth; the RBF kernel enforces this. The length-scale ℓ=1.0\ell = 1.0 means the functions vary on a scale of roughly 1 unit in xx. Data will prune this infinite family down to the functions consistent with our observations.

Hyperparameter Optimisation: Marginal Log-Likelihood

How did we arrive at σf=1.27\sigma_f = 1.27 and ℓ=1.0\ell = 1.0? The original code optimises the marginal log-likelihood (Ebden Eq. 10):

log⁡p(y∣x,θ)=−12y⊤K−1y−12log⁡∣K∣−n2log⁡2π\log p(\mathbf{y} \mid \mathbf{x}, \boldsymbol{\theta}) = -\frac{1}{2}\mathbf{y}^\top K^{-1}\mathbf{y} - \frac{1}{2}\log|K| - \frac{n}{2}\log 2\pi

This balances three terms:

  • Data fit (−12y⊤K−1y-\frac{1}{2}\mathbf{y}^\top K^{-1}\mathbf{y}): how well the model explains the observations
  • Complexity penalty (−12log⁡∣K∣-\frac{1}{2}\log|K|): penalises overly flexible models (Occam's razor)
  • Normalisation (−n2log⁡2π-\frac{n}{2}\log 2\pi): constant

This is maximum likelihood estimation applied to the marginal likelihood: "marginal" because we've integrated out the function values, leaving only the hyperparameters.

from scipy.optimize import minimize

def neg_log_marginal_likelihood(params, X_train, y_train, sigma_n):
    """Negative log marginal likelihood (Ebden Eq. 10)."""
    sigma_f, l = np.abs(params)  # ensure positive
    n = len(X_train)
    K = kernel_matrix(X_train, X_train, sigma_f, l) + sigma_n**2 * np.eye(n)

    log_lik = (-0.5 * y_train @ inv(K) @ y_train
               - 0.5 * np.log(det(K))
               - n / 2 * np.log(2 * np.pi))
    return -log_lik

# Optimise sigma_f and l, keeping sigma_n=0.3 fixed (matching original code)
result = minimize(
    neg_log_marginal_likelihood,
    x0=[0.1, 0.5],  # initial values from original code
    args=(X_train, y_train, 0.3),
    method='Nelder-Mead'
)
sigma_f_opt, l_opt = np.abs(result.x)
print(f'Optimised: sigma_f={sigma_f_opt:.2f}, l={l_opt:.2f}')
# Optimised: sigma_f=1.34, l=1.04

The log-likelihood landscape shows a clear optimum:

Contour plot of log marginal likelihood over sigma_f and l, with the optimum marked at (1.34, 1.04).

The optimiser starts at (σf,ℓ)=(0.1,0.5)(\sigma_f, \ell) = (0.1, 0.5) and converges to (1.34,1.04)(1.34, 1.04), close to the tutorial's reported values of (1.27,1.0)(1.27, 1.0). The small difference comes from Nelder-Mead finding a slightly different local optimum; the negative log-likelihoods differ by only 0.004. The complexity penalty prevents overfitting; without it, the model would shrink ℓ\ell to interpolate every noisy observation exactly.

A fully Bayesian approach would place priors on σf\sigma_f and ℓ\ell and integrate over them using MCMC. For most applications, the point estimate from marginal likelihood optimisation works well.

Validating with sklearn

Let's confirm our from-scratch implementation matches scikit-learn:

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel

kernel = 1.27**2 * RBF(length_scale=1.0) + WhiteKernel(noise_level=0.3**2)
gpr = GaussianProcessRegressor(kernel=kernel, optimizer=None, alpha=0)

gpr.fit(X_train.reshape(-1, 1), y_train)
mu_sk, std_sk = gpr.predict(X_test.reshape(-1, 1), return_std=True)

print(f'Max difference in mean: {np.max(np.abs(mu - mu_sk)):.6f}')
# Max difference in mean: ~1e-14 (numerical precision)

Our NumPy implementation produces identical results.

When NOT to Use Gaussian Processes

GPs have a fundamental limitation: cubic scaling. Computing K−1K^{-1} is O(n3)O(n^3) in the number of training points. With 100 points it's instant; with 1,000 it takes seconds; with 10,000 it becomes impractical.

For large datasets, consider:

  • Sparse GP approximations: use a subset of inducing points to approximate the full GP
  • Random Fourier features: approximate the kernel with explicit feature maps
  • Relevance Vector Machines: Tipping's Bayesian alternative that naturally produces sparse kernel solutions
  • Neural networks: scale linearly in data but lose calibrated uncertainty

The RBF kernel also assumes stationarity: the same smoothness everywhere. If your function is smooth in one region and jagged in another, you'd need a non-stationary kernel or a different model.

Where This Comes From

Rasmussen & Williams (2006)

The definitive reference is Rasmussen, C.E. & Williams, C.K.I. (2006) Gaussian Processes for Machine Learning, MIT Press. The full text is available free online.

They define a Gaussian process formally:

"A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution."

A GP is fully specified by its mean function m(x)m(x) and covariance function k(x,x′)k(x, x'):

f(x)∼GP ⁣(m(x), k(x,x′))f(x) \sim \mathcal{GP}\!\left(m(x),\, k(x, x')\right)

We set m(x)=0m(x) = 0 (standard practice: the kernel handles everything), so the covariance function alone defines the GP. This is the function-space view: we're placing a prior directly on functions, not on parameters.

Rasmussen & Williams also present the weight-space view (Chapter 2.1). In Bayesian linear regression with basis functions ϕ(x)\phi(x), the predictive distribution is:

f∗∣x∗,X,y∼N ⁣(ϕ∗⊤ΣpΦ(K)−1y,  ϕ∗⊤Σpϕ∗)f_* \mid \mathbf{x}_*, X, \mathbf{y} \sim \mathcal{N}\!\left(\phi_*^\top \Sigma_p \Phi (K)^{-1} \mathbf{y},\; \phi_*^\top \Sigma_p \phi_*\right)

As the number of basis functions grows to infinity, this converges to the GP formulation; the function-space and weight-space views are equivalent. This connects GPs to neural networks: Neal (1996) showed that a single-hidden-layer neural network with infinitely many hidden units converges to a GP.

Ebden (2008): Our Implementation Reference

Our implementation follows Ebden, M. (2008) "Gaussian Processes for Regression: A Quick Introduction", which maps directly to our code:

Tutorial EquationOur CodeWhat it computes
Eq. (1): RBF kernelrbf_kernel()k(x,x′)=σf2exp⁡(−(x−x′)2/2ℓ2)k(x, x') = \sigma_f^2 \exp(-(x-x')^2 / 2\ell^2)
Eq. (4): KK matrixkernel_matrix() + sigma_n^2 * ITraining covariance
Eq. (5): K∗K_* and K∗∗K_{**}kernel_matrix() callsTest-train and test-test covariance
Eq. (8): Predictive meanK_s @ K_inv @ y_trainyˉ∗=K∗K−1y\bar{y}_* = K_* K^{-1}\mathbf{y}
Eq. (9): Predictive varianceK_ss - K_s @ K_inv @ K_s.TK∗∗−K∗K−1K∗⊤K_{**} - K_* K^{-1} K_*^\top
Eq. (10): Log-likelihoodneg_log_marginal_likelihood()Hyperparameter optimisation

Historical Context

The idea of using stochastic processes for interpolation has deep roots:

  • Kolmogorov (1941) and Wiener (1949): optimal linear prediction theory
  • Matheron (1963): "kriging" in geostatistics, named after D.G. Krige's mining work in South Africa
  • O'Hagan (1978): formalised the Bayesian interpretation
  • MacKay (1998): introduced GPs to the machine learning community
  • Neal (1996): proved the GP-neural network connection
  • Rasmussen & Williams (2006): the modern comprehensive treatment

Kriging and GP regression are mathematically identical; the geostatistics and machine learning communities developed the same ideas independently, using different vocabulary (variograms vs kernels, kriging variance vs posterior variance).

Further Reading

Try It Yourself

The interactive notebook includes exercises:

  1. Different kernels: Implement the Matern 5/2 kernel and compare its predictions to the RBF. How do the confidence bands differ?
  2. Multi-dimensional inputs: Extend the GP to 2D inputs. The kernel becomes k(x,x′)=σf2exp⁡(−∥x−x′∥2/2ℓ2)k(\mathbf{x}, \mathbf{x}') = \sigma_f^2 \exp(-\|\mathbf{x} - \mathbf{x}'\|^2 / 2\ell^2).
  3. Noisy function: Generate data from y=sin⁡(x)+ϵy = \sin(x) + \epsilon with ϵ∼N(0,0.22)\epsilon \sim \mathcal{N}(0, 0.2^2). Fit a GP and observe how the confidence band captures the noise.
  4. Bayesian optimisation: Use your GP to implement a simple acquisition function (Expected Improvement) and optimise a 1D black-box function.

Interactive Tools

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions