You're comparing model settings, and each training run takes long enough that you cannot try every possibility. A handful of completed runs gives you some evidence, but leaves large gaps between the settings you tested. Gaussian process regression offers a way to estimate values in those gaps while also describing uncertainty. This makes it useful when the next measurement or experiment needs to be chosen carefully.
We fit a curve to turn gaps between observations into a prediction and variance. At input 0.2, the GP predicts about 0.98 with variance 0.21. Kernel conditioning links observations to that estimate, following the model used by our hyperparameter optimisation example.
This is regression with built-in uncertainty quantification. Unlike fitting a line or a polynomial, a GP doesn't commit to a fixed functional form. Instead, it defines a distribution over functions and lets the data narrow it down. The result is a smooth prediction with a confidence band that widens where data is sparse and tightens where it's dense.
By the end of this post, you'll implement GP regression from scratch with NumPy, understand how the kernel function encodes your assumptions about smoothness, and see exactly why GPs power Bayesian optimisation.
Let's Build It
Click the badge to open the interactive notebook:
Watch how the GP posterior sharpens as we feed it observations one by one. The shaded region represents 95% confidence, and it collapses around each data point:

Here's the complete implementation. We'll use 6 noisy observations from Ebden's GP tutorial (2008) and predict across a dense grid:
import numpy as np
from numpy.linalg import inv, det
import matplotlib.pyplot as plt
# --- The Kernel ---
def rbf_kernel(x1, x2, sigma_f, l):
"""Squared exponential (RBF) kernel."""
return sigma_f**2 * np.exp(-0.5 * (x1 - x2)**2 / l**2)
def kernel_matrix(X1, X2, sigma_f, l):
"""Compute kernel matrix between two sets of points."""
return np.array([[rbf_kernel(a, b, sigma_f, l) for b in X2] for a in X1])
# --- GP Prediction (Ebden Eq. 8 & 9) ---
def gp_predict(X_train, y_train, X_test, sigma_f, l, sigma_n):
"""Predict mean and variance at test points."""
n = len(X_train)
K = kernel_matrix(X_train, X_train, sigma_f, l) + sigma_n**2 * np.eye(n)
K_s = kernel_matrix(X_test, X_train, sigma_f, l)
K_ss = kernel_matrix(X_test, X_test, sigma_f, l) + sigma_n**2 * np.eye(len(X_test))
K_inv = inv(K)
mu = K_s @ K_inv @ y_train
cov = K_ss - K_s @ K_inv @ K_s.T
return mu, cov
# --- Data from Ebden (2008) tutorial ---
X_train = np.array([-1.50, -1.00, -0.75, -0.40, -0.25, 0.00])
y_train = 0.55 * np.array([-3.0, -2.0, -0.6, 0.4, 1.0, 1.6])
# Hyperparameters (optimised values from the tutorial: sigma_f=1.27, l=1.0)
sigma_n = 0.3 # observation noise (known from error bars)
sigma_f = 1.27 # signal standard deviation (optimised)
l = 1.0 # length-scale (optimised)
# Predict on a dense grid
X_test = np.linspace(-2.0, 1.0, 200)
mu, cov = gp_predict(X_train, y_train, X_test, sigma_f, l, sigma_n)
std = np.sqrt(np.diag(cov))
# Plot
fig, ax = plt.subplots(figsize=(10, 5))
ax.fill_between(X_test, mu - 1.96 * std, mu + 1.96 * std,
alpha=0.2, color='tab:blue', label='95% confidence')
ax.plot(X_test, mu, 'tab:blue', linewidth=2, label='GP mean')
ax.errorbar(X_train, y_train, yerr=sigma_n, fmt='ro', capsize=4,
markersize=6, label='Training data')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend()
plt.show()

The blue line is our best estimate. The shaded band is the 95% confidence interval: wide where we have no data, narrow near the observations. At , the GP predicts with variance 0.21, closely matching Ebden's worked example (, ).
What Just Happened?
Three ingredients make GP regression work: a kernel function that encodes our smoothness assumptions, covariance matrices that capture relationships between all points, and conditioning that turns the prior into a posterior.
The Kernel: Encoding Smoothness
The squared exponential (RBF) kernel measures similarity between inputs:
Think of it as answering: "If I know the function value at , how much does that tell me about ?"
- Close together (): (strong correlation, they "see" each other)
- Far apart (): (independent, no information shared)
The two hyperparameters control different things:
- (signal std): the typical amplitude of the function. Higher means larger vertical swings.
- (length-scale): how far you need to move in before the function value changes significantly. Short gives wiggly functions; long gives smooth ones.
There's also (noise std), capturing measurement noise: each observation relates to the true function via .
The effect of the length-scale is dramatic:

With , the GP tries to wiggle through every point; it over-fits. With , it's so smooth it can't follow the data's trend; it under-fits. The optimised strikes the right balance.
Building the Covariance Matrices
To make predictions, we compute three kernel matrices:
, between all training points (Ebden Eq. 4):
The on the diagonal accounts for observation noise. Off-diagonal entries capture how correlated two training points are.
, between test and training points (Ebden Eq. 5):
, between test points:
In our code, kernel_matrix computes these. The + sigma_n**2 * np.eye(n) adds noise to the diagonal.
The Prediction Equations
The GP assumption is that training outputs and test outputs are jointly Gaussian:
Conditioning on the observed gives the posterior, also Gaussian (this is the beauty of Gaussians: conditioning preserves Gaussianity):
The mean is a weighted combination of the training outputs, where the weights come from how correlated is with each training point. The variance shrinks where has large entries (near training data) and expands where is far from any observation.
This is Bayesian inference in action: the prior (encoded by the kernel) gets updated by the data to produce a posterior that's both a prediction and an honest assessment of uncertainty.
Going Deeper
GP Prior: What the Kernel Believes Before Seeing Data
Before any observations, the GP prior defines a distribution over functions. We can sample from it by drawing from where is the kernel matrix evaluated on a grid:
X_grid = np.linspace(-2, 2, 200)
K_prior = kernel_matrix(X_grid, X_grid, sigma_f=1.27, l=1.0)
K_prior += 1e-8 * np.eye(len(X_grid)) # numerical stability
# Draw 3 random functions from the prior
np.random.seed(42)
samples = np.random.multivariate_normal(np.zeros(len(X_grid)), K_prior, size=5)
fig, ax = plt.subplots(figsize=(10, 5))
for i in range(5):
ax.plot(X_grid, samples[i], label=f'Sample {i+1}')
ax.fill_between(X_grid, -1.96 * 1.27, 1.96 * 1.27, alpha=0.1, color='grey',
label='95% prior band')
ax.set_xlabel('x')
ax.set_ylabel('f(x)')
ax.legend()
plt.show()

Every sample is smooth; the RBF kernel enforces this. The length-scale means the functions vary on a scale of roughly 1 unit in . Data will prune this infinite family down to the functions consistent with our observations.
Hyperparameter Optimisation: Marginal Log-Likelihood
How did we arrive at and ? The original code optimises the marginal log-likelihood (Ebden Eq. 10):
This balances three terms:
- Data fit (): how well the model explains the observations
- Complexity penalty (): penalises overly flexible models (Occam's razor)
- Normalisation (): constant
This is maximum likelihood estimation applied to the marginal likelihood: "marginal" because we've integrated out the function values, leaving only the hyperparameters.
from scipy.optimize import minimize
def neg_log_marginal_likelihood(params, X_train, y_train, sigma_n):
"""Negative log marginal likelihood (Ebden Eq. 10)."""
sigma_f, l = np.abs(params) # ensure positive
n = len(X_train)
K = kernel_matrix(X_train, X_train, sigma_f, l) + sigma_n**2 * np.eye(n)
log_lik = (-0.5 * y_train @ inv(K) @ y_train
- 0.5 * np.log(det(K))
- n / 2 * np.log(2 * np.pi))
return -log_lik
# Optimise sigma_f and l, keeping sigma_n=0.3 fixed (matching original code)
result = minimize(
neg_log_marginal_likelihood,
x0=[0.1, 0.5], # initial values from original code
args=(X_train, y_train, 0.3),
method='Nelder-Mead'
)
sigma_f_opt, l_opt = np.abs(result.x)
print(f'Optimised: sigma_f={sigma_f_opt:.2f}, l={l_opt:.2f}')
# Optimised: sigma_f=1.34, l=1.04
The log-likelihood landscape shows a clear optimum:

The optimiser starts at and converges to , close to the tutorial's reported values of . The small difference comes from Nelder-Mead finding a slightly different local optimum; the negative log-likelihoods differ by only 0.004. The complexity penalty prevents overfitting; without it, the model would shrink to interpolate every noisy observation exactly.
A fully Bayesian approach would place priors on and and integrate over them using MCMC. For most applications, the point estimate from marginal likelihood optimisation works well.
Validating with sklearn
Let's confirm our from-scratch implementation matches scikit-learn:
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel
kernel = 1.27**2 * RBF(length_scale=1.0) + WhiteKernel(noise_level=0.3**2)
gpr = GaussianProcessRegressor(kernel=kernel, optimizer=None, alpha=0)
gpr.fit(X_train.reshape(-1, 1), y_train)
mu_sk, std_sk = gpr.predict(X_test.reshape(-1, 1), return_std=True)
print(f'Max difference in mean: {np.max(np.abs(mu - mu_sk)):.6f}')
# Max difference in mean: ~1e-14 (numerical precision)
Our NumPy implementation produces identical results.
When NOT to Use Gaussian Processes
GPs have a fundamental limitation: cubic scaling. Computing is in the number of training points. With 100 points it's instant; with 1,000 it takes seconds; with 10,000 it becomes impractical.
For large datasets, consider:
- Sparse GP approximations: use a subset of inducing points to approximate the full GP
- Random Fourier features: approximate the kernel with explicit feature maps
- Relevance Vector Machines: Tipping's Bayesian alternative that naturally produces sparse kernel solutions
- Neural networks: scale linearly in data but lose calibrated uncertainty
The RBF kernel also assumes stationarity: the same smoothness everywhere. If your function is smooth in one region and jagged in another, you'd need a non-stationary kernel or a different model.
Where This Comes From
Rasmussen & Williams (2006)
The definitive reference is Rasmussen, C.E. & Williams, C.K.I. (2006) Gaussian Processes for Machine Learning, MIT Press. The full text is available free online.
They define a Gaussian process formally:
"A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution."
A GP is fully specified by its mean function and covariance function :
We set (standard practice: the kernel handles everything), so the covariance function alone defines the GP. This is the function-space view: we're placing a prior directly on functions, not on parameters.
Rasmussen & Williams also present the weight-space view (Chapter 2.1). In Bayesian linear regression with basis functions , the predictive distribution is:
As the number of basis functions grows to infinity, this converges to the GP formulation; the function-space and weight-space views are equivalent. This connects GPs to neural networks: Neal (1996) showed that a single-hidden-layer neural network with infinitely many hidden units converges to a GP.
Ebden (2008): Our Implementation Reference
Our implementation follows Ebden, M. (2008) "Gaussian Processes for Regression: A Quick Introduction", which maps directly to our code:
| Tutorial Equation | Our Code | What it computes |
|---|---|---|
| Eq. (1): RBF kernel | rbf_kernel() | |
| Eq. (4): matrix | kernel_matrix() + sigma_n^2 * I | Training covariance |
| Eq. (5): and | kernel_matrix() calls | Test-train and test-test covariance |
| Eq. (8): Predictive mean | K_s @ K_inv @ y_train | |
| Eq. (9): Predictive variance | K_ss - K_s @ K_inv @ K_s.T | |
| Eq. (10): Log-likelihood | neg_log_marginal_likelihood() | Hyperparameter optimisation |
Historical Context
The idea of using stochastic processes for interpolation has deep roots:
- Kolmogorov (1941) and Wiener (1949): optimal linear prediction theory
- Matheron (1963): "kriging" in geostatistics, named after D.G. Krige's mining work in South Africa
- O'Hagan (1978): formalised the Bayesian interpretation
- MacKay (1998): introduced GPs to the machine learning community
- Neal (1996): proved the GP-neural network connection
- Rasmussen & Williams (2006): the modern comprehensive treatment
Kriging and GP regression are mathematically identical; the geostatistics and machine learning communities developed the same ideas independently, using different vocabulary (variograms vs kernels, kriging variance vs posterior variance).
Further Reading
For estimating a treatment effect with learned nuisance models, the double machine learning tutorial works through residualisation and cross-fitting.
Rasmussen & Williams (2006): The full textbook, free online. Start with Chapter 2.
Ebden (2008): The concise tutorial our code follows
MacKay (1998): "Introduction to Gaussian Processes", a shorter, more accessible introduction
Our Bayesian optimisation post: see GPs in action as the surrogate model
Try It Yourself
The interactive notebook includes exercises:
- Different kernels: Implement the Matern 5/2 kernel and compare its predictions to the RBF. How do the confidence bands differ?
- Multi-dimensional inputs: Extend the GP to 2D inputs. The kernel becomes .
- Noisy function: Generate data from with . Fit a GP and observe how the confidence band captures the noise.
- Bayesian optimisation: Use your GP to implement a simple acquisition function (Expected Improvement) and optimise a 1D black-box function.
Interactive Tools
- GP Regression Playground: Fit Gaussian processes to your own data and experiment with kernels in the browser
Related Posts
- Hyperparameter Optimization: Grid vs Random vs Bayesian: See GPs as the surrogate model in Bayesian optimisation
- From MLE to Bayesian Inference: The Bayesian framework that underpins GP regression
- MCMC Island Hopping: Understanding Metropolis-Hastings: An alternative to point estimates for GP hyperparameters
- Hamiltonian Neural Networks: A different prior on smooth dynamics: GPs encode it in a kernel, Hamiltonian nets in a learned energy function
- Time Series as Dynamical Systems: Delay Embedding: A GP models the series as a smooth function of time; delay embedding models it as a trajectory through reconstructed state space
- Conformal Prediction: 90% Coverage, 29% Where It Counts: The GP's predictive band measured against conformal intervals on the same heteroscedastic data, where only the conformal coverage is guaranteed
- Kolmogorov-Arnold Networks in Python: A Curve on Every Edge: Another basis-function view of fitting curves, with a learned B-spline on every edge of a network







