Your network has calculated a gradient, but it still needs a rule for turning that information into a change to its weights. A step that is too large may overshoot, while small steps can make progress slow. The optimiser chooses those updates, and different choices can produce different training paths even with the same starting point.
We trace how stored gradients change each update by building SGD, momentum and Adam in NumPy. After 2,000 Rosenbrock steps, SGD remains 1.72 from the minimum while Adam is 0.0015 away. PyTorch checks and a CNN trained on MNIST test those implementations.
Let's Build It
First, let's watch all four optimisers race across the same loss surface. Click the badge to run this yourself:

Here is the complete code. We define all four optimisers and let them minimise the Rosenbrock function, a classic test surface with a narrow curved valley leading to the minimum at (1, 1).
import numpy as np
def rosenbrock(x, y):
"""Rosenbrock function: narrow curved valley, minimum at (1, 1)."""
return (1 - x)**2 + 100 * (y - x**2)**2
def rosenbrock_grad(x, y):
"""Gradient of the Rosenbrock function."""
dx = -2 * (1 - x) - 400 * x * (y - x**2)
dy = 200 * (y - x**2)
return np.array([dx, dy])
# --- SGD ---
def sgd(grad_fn, x0, lr=0.0002, steps=2000):
path = [x0.copy()]
x = x0.copy()
for _ in range(steps):
g = grad_fn(*x)
x -= lr * g
path.append(x.copy())
return np.array(path)
# --- SGD + Momentum ---
def sgd_momentum(grad_fn, x0, lr=0.0002, mom=0.9, steps=2000):
path = [x0.copy()]
x = x0.copy()
v = np.zeros_like(x)
for _ in range(steps):
g = grad_fn(*x)
v = mom * v + g
x -= lr * v
path.append(x.copy())
return np.array(path)
# --- RMSProp ---
def rmsprop(grad_fn, x0, lr=0.001, beta=0.99, eps=1e-8, steps=2000):
path = [x0.copy()]
x = x0.copy()
v = np.zeros_like(x)
for _ in range(steps):
g = grad_fn(*x)
v = beta * v + (1 - beta) * g**2
x -= lr * g / (np.sqrt(v) + eps)
path.append(x.copy())
return np.array(path)
# --- Adam ---
def adam(grad_fn, x0, lr=0.005, beta1=0.9, beta2=0.999, eps=1e-8, steps=2000):
path = [x0.copy()]
x = x0.copy()
m = np.zeros_like(x) # first moment (mean of gradients)
v = np.zeros_like(x) # second moment (mean of squared gradients)
for t in range(1, steps + 1):
g = grad_fn(*x)
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * g**2
m_hat = m / (1 - beta1**t) # bias correction
v_hat = v / (1 - beta2**t) # bias correction
x -= lr * m_hat / (np.sqrt(v_hat) + eps)
path.append(x.copy())
return np.array(path)
# Run all four from the same starting point
x0 = np.array([-1.0, 1.0])
path_sgd = sgd(rosenbrock_grad, x0, lr=0.0002, steps=2000)
path_mom = sgd_momentum(rosenbrock_grad, x0, lr=0.0002, mom=0.9, steps=2000)
path_rms = rmsprop(rosenbrock_grad, x0, lr=0.001, steps=2000)
path_adam = adam(rosenbrock_grad, x0, lr=0.005, steps=2000)
# Global minimum is at (1, 1)
for name, path in [("SGD", path_sgd), ("Momentum", path_mom),
("RMSProp", path_rms), ("Adam", path_adam)]:
dist = np.sqrt((path[-1][0] - 1)**2 + (path[-1][1] - 1)**2)
print(f"{name:10s} final=({path[-1][0]:.4f}, {path[-1][1]:.4f}), dist to min={dist:.4f}")
SGD final=(-0.5920, 0.3583), dist to min=1.7165
Momentum final=( 0.9071, 0.8224), dist to min=0.2004
RMSProp final=( 0.8133, 0.6607), dist to min=0.3872
Adam final=( 0.9994, 0.9987), dist to min=0.0015
After 2,000 steps, vanilla SGD is still far from the minimum. Momentum reaches the valley and gets close. RMSProp adapts its learning rate per dimension but gets stuck in the curve. Adam combines both techniques and essentially solves it. Now let's understand why.
What Just Happened?
Vanilla SGD: The Simplest Optimiser
Stochastic gradient descent is the foundation of all neural network training. The update rule is one line:
Where are the parameters, is the learning rate, and is the gradient of the loss. Compute the gradient, take a step in the opposite direction, repeat.
def sgd_step(params, grads, lr):
"""One step of vanilla SGD. This is the entire algorithm."""
for p, g in zip(params, grads):
p -= lr * g
The problem is that SGD treats every direction equally. If the loss surface is a narrow valley (steep walls, gentle slope along the floor), SGD oscillates across the walls while making slow progress along the floor. This is exactly what happens on the Beale function.

Momentum: Remembering Past Gradients
Momentum fixes the oscillation problem by accumulating a running average of past gradients. Think of a ball rolling down a hill: it builds up speed in the consistent direction while the back-and-forth oscillations cancel out.
The velocity accumulates gradients over time. The momentum coefficient (typically 0.9) controls how much history to keep. With , the effective window is roughly the last 10 gradients.
def sgd_momentum_step(params, grads, velocities, lr, mom=0.9):
"""SGD with momentum. Velocity accumulates past gradients."""
for p, g, v in zip(params, grads, velocities):
v[:] = mom * v + g
p -= lr * v
This is the approach used in the fast.ai course, where Jeremy Howard implements it as a StatefulOptimizer with an AverageGrad statistic tracking the running average.
Dampened Momentum vs Classical Momentum
There are two conventions for momentum. The classical form (used above and in the fast.ai notebooks) computes . The dampened form (used in Adam) computes . The dampening factor keeps the running average at the same scale as the original gradients, which matters for bias correction.
# Classical momentum (fast.ai/PyTorch SGD default)
v = mom * v + grad
# Dampened momentum (used inside Adam)
v = mom * v + (1 - mom) * grad
With classical momentum, the running average is roughly ; with dampening, it is roughly . The dampened form is easier to reason about because stays on the same scale as the gradients regardless of .
Exponential Moving Average and Debiasing
Both momentum and Adam use exponential moving averages (EMAs). An EMA with coefficient gives exponentially decaying weight to older values:
There is a problem at the start: , so the first few estimates are biased towards zero. The fix is bias correction, which divides by :
This term starts near 0 when is small (strong correction) and approaches 1 as grows (no correction needed).
def ema_with_debiasing(values, beta=0.9):
"""Exponential moving average with bias correction."""
avg = 0.0
corrected = []
for t, x in enumerate(values, 1):
avg = beta * avg + (1 - beta) * x
corrected.append(avg / (1 - beta**t))
return corrected

The fast.ai notebook (dl2/09_optimizers.ipynb) has an excellent visualisation of this: with high , the undebiased average starts far too low and takes many steps to catch up. Debiasing fixes the initial estimates without affecting the long-run behaviour.
RMSProp: Adaptive Learning Rates
SGD and momentum use the same learning rate for every parameter. But some parameters might have consistently large gradients (they need smaller steps) while others have tiny gradients (they need larger steps). RMSProp adapts the learning rate per parameter by dividing by the root mean square of recent gradients.
Parameters with large gradients get divided by a large number (smaller effective step). Parameters with small gradients get divided by a small number (larger effective step). This is why RMSProp reaches the minimum on the Beale function even though vanilla SGD cannot.
def rmsprop_step(params, grads, sq_avgs, lr, beta=0.9, eps=1e-8):
"""RMSProp: adapt learning rate by gradient magnitude."""
for p, g, v in zip(params, grads, sq_avgs):
v[:] = beta * v + (1 - beta) * g**2
p -= lr * g / (np.sqrt(v) + eps)
RMSProp was proposed by Geoffrey Hinton in lecture 6e of his Coursera course (2012), never formally published in a paper. It remains widely used, especially in reinforcement learning.
Adam: The Best of Both Worlds
Adam (Adaptive Moment Estimation) combines momentum and RMSProp. It maintains two running averages: (first moment, like momentum) and (second moment, like RMSProp), both with bias correction.
Here is the complete implementation, matching the fast.ai notebook's adam_step:
class AdamNumPy:
"""Adam optimiser from scratch."""
def __init__(self, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8):
self.lr = lr
self.beta1 = beta1
self.beta2 = beta2
self.eps = eps
self.m = {} # first moments
self.v = {} # second moments
self.t = 0
def step(self, params, grads):
self.t += 1
for i, (p, g) in enumerate(zip(params, grads)):
if i not in self.m:
self.m[i] = np.zeros_like(p)
self.v[i] = np.zeros_like(p)
self.m[i] = self.beta1 * self.m[i] + (1 - self.beta1) * g
self.v[i] = self.beta2 * self.v[i] + (1 - self.beta2) * g**2
m_hat = self.m[i] / (1 - self.beta1**self.t)
v_hat = self.v[i] / (1 - self.beta2**self.t)
p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)
Verifying Against PyTorch
Let's confirm our NumPy Adam produces the same parameter updates as torch.optim.Adam:
import torch
# Shared setup
np.random.seed(42)
w_np = np.array([1.0, -2.0, 0.5])
g_np = np.array([0.1, -0.3, 0.2])
# NumPy Adam
adam_np = AdamNumPy(lr=0.001)
w_test = w_np.copy()
adam_np.step([w_test], [g_np.copy()])
# PyTorch Adam
w_pt = torch.tensor(w_np, requires_grad=True)
opt = torch.optim.Adam([w_pt], lr=0.001)
w_pt.grad = torch.tensor(g_np)
opt.step()
print(f"NumPy: {w_test}")
print(f"PyTorch: {w_pt.detach().numpy()}")
print(f"Match: {np.allclose(w_test, w_pt.detach().numpy(), atol=1e-7)}")
NumPy: [ 0.999 -1.999 0.499]
PyTorch: [ 0.999 -1.999 0.499]
Match: True
Going Deeper
Weight Decay vs L2 Regularisation
The fast.ai notebook makes an important distinction: L2 regularisation adds to the gradient, while decoupled weight decay subtracts directly from the parameters.
# L2 regularisation: modifies the gradient
grad += wd * param
# Decoupled weight decay: modifies the parameter directly
param *= (1 - lr * wd)
For vanilla SGD, these are mathematically equivalent. But for Adam, they are not. The AdamW paper (Loshchilov & Hutter, 2019) showed that decoupled weight decay generalises better with Adam because the adaptive learning rate doesn't interfere with the regularisation strength. PyTorch's torch.optim.AdamW implements this, and it is the default in most modern training recipes.

Training a CNN: SGD vs Adam in Practice
Let's benchmark our optimisers on a real task: training the SimpleCNN from our previous post on MNIST.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))
])
train_data = datasets.MNIST('data', train=True, download=True, transform=transform)
test_data = datasets.MNIST('data', train=False, transform=transform)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=1000)
class SimpleCNN(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 8, kernel_size=5, padding=2, stride=2)
self.conv2 = nn.Conv2d(8, 16, kernel_size=3, padding=1, stride=2)
self.conv3 = nn.Conv2d(16, 32, kernel_size=3, padding=1, stride=2)
self.fc = nn.Linear(32 * 4 * 4, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
x = F.relu(self.conv2(x))
x = F.relu(self.conv3(x))
x = x.view(x.size(0), -1)
return self.fc(x)
def train_and_evaluate(opt_class, opt_kwargs, epochs=5, seed=42):
torch.manual_seed(seed)
model = SimpleCNN()
optimiser = opt_class(model.parameters(), **opt_kwargs)
results = []
for epoch in range(epochs):
model.train()
for images, labels in train_loader:
loss = F.cross_entropy(model(images), labels)
optimiser.zero_grad()
loss.backward()
optimiser.step()
model.eval()
correct = 0
with torch.no_grad():
for images, labels in test_loader:
correct += (model(images).argmax(1) == labels).sum().item()
results.append(correct / len(test_data) * 100)
return results
configs = {
"SGD (lr=0.01)": (torch.optim.SGD, {"lr": 0.01}),
"SGD+Momentum (lr=0.01)": (torch.optim.SGD, {"lr": 0.01, "momentum": 0.9}),
"Adam (lr=0.001)": (torch.optim.Adam, {"lr": 0.001}),
"AdamW (lr=0.001)": (torch.optim.AdamW, {"lr": 0.001, "weight_decay": 0.01}),
}
for name, (opt_cls, kwargs) in configs.items():
acc = train_and_evaluate(opt_cls, kwargs)
print(f"{name:25s} -> {[f'{a:.1f}%' for a in acc]}")
SGD (lr=0.01) -> ['88.6%', '92.5%', '94.4%', '95.7%', '95.5%']
SGD+Momentum (lr=0.01) -> ['97.3%', '98.1%', '98.4%', '98.5%', '98.5%']
Adam (lr=0.001) -> ['96.7%', '97.7%', '98.2%', '98.6%', '98.4%']
AdamW (lr=0.001) -> ['96.7%', '97.7%', '98.2%', '98.5%', '98.5%']

Vanilla SGD is dramatically slower, reaching only 95.5% after 5 epochs. Momentum transforms the picture: 97.3% after just one epoch, matching Adam's convergence speed. Adam and AdamW reach comparable final accuracy, with momentum matching AdamW at epoch 5 (98.5%) and edging past Adam (98.4%) thanks to its stronger gradient signal at this learning rate. The takeaway: momentum is the most impactful single improvement over vanilla SGD.
Choosing an Optimiser
| Optimiser | When to use | Typical LR |
|---|---|---|
| SGD | Large-scale vision (with careful LR scheduling) | 0.01-0.1 |
| SGD + Momentum | Default for CNNs when you want maximum generalisation | 0.01-0.1 |
| Adam | Fast prototyping, NLP, GANs, any new project | 0.0001-0.001 |
| AdamW | Production training with regularisation | 0.0001-0.001 |
A common observation in deep learning: Adam converges faster but SGD+momentum can generalise better if you invest time in learning rate scheduling. Many state-of-the-art vision models use SGD with cosine annealing. For most practitioners starting a new project, Adam is the right default.
When Optimisers Fail
Learning rate too high. The loss explodes or oscillates wildly. Adam is more forgiving than SGD because the adaptive denominator acts as an automatic brake, but even Adam diverges if the learning rate is absurdly high.
Learning rate too low. Training is stable but painfully slow. The loss decreases but never reaches a good minimum. This is especially common with vanilla SGD.
Gradient vanishing/exploding. If gradients are near zero (vanishing) or enormous (exploding), no optimiser can help. Fix the architecture first (batch normalisation, proper initialisation, skip connections).
Loss plateaus. The optimiser gets stuck on a flat region of the loss surface. Momentum helps because accumulated velocity carries it through flat spots. Learning rate warmup (starting low, then increasing) also helps by letting Adam's moment estimates stabilise before taking large steps.
Where This Comes From
The Robbins-Monro Framework (1951)
Stochastic gradient descent traces back to Herbert Robbins and Sutton Monro's 1951 paper, "A Stochastic Approximation Method." They proved that iterative updates using noisy gradient estimates converge to the optimum under mild conditions (the learning rate must decrease over time such that and ).
"Given a function M(x) for which M(theta) = alpha, where alpha is known but M is unknown, we propose a method of estimating theta by successive approximations." -- Robbins & Monro (1951)
Modern SGD relaxes the decreasing learning rate requirement by using mini-batches (which reduce gradient noise) and learning rate schedules.
Momentum: Polyak (1964) and Nesterov (1983)
Boris Polyak introduced the heavy ball method in 1964, adding a momentum term to gradient descent. Yurii Nesterov improved it in 1983 with "accelerated gradient descent," which evaluates the gradient at the look-ahead position rather than the current position. Nesterov momentum converges faster for convex problems, and PyTorch supports it via nesterov=True in torch.optim.SGD.
Adam: Kingma & Ba (2014)
Adam was introduced by Diederik Kingma and Jimmy Ba in their 2014 paper, "Adam: A Method for Stochastic Optimization". The key insight was combining the first moment (momentum) with the second moment (RMSProp) and adding bias correction for both.
"We propose Adam, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients." -- Kingma & Ba (2014)
Adam quickly became the default optimiser for deep learning. Its robustness to hyperparameter choices (the defaults , , work well across a wide range of problems) made it the practical choice for most researchers.
Beyond Adam
The fast.ai notebook implements LAMB (Layer-wise Adaptive Moments for Batch training), which scales Adam's update by the ratio of parameter norm to update norm. This enables training with very large batch sizes. Other recent variants include:
- AdaFactor (Shazeer & Stern, 2018): Reduces memory by factorising the second moment matrix
- LARS (You et al., 2017): Layer-wise adaptive rates for large-batch SGD
- Lion (Chen et al., 2023): A simpler optimiser found by program search that uses only the sign of the momentum
Further Reading
The weight initialisation experiments examine activation and gradient behaviour before training.
The Adam paper: Kingma, D.P. & Ba, J. (2014). "Adam: A Method for Stochastic Optimization." Read sections 2-3 for the algorithm and bias correction derivation.
AdamW: Loshchilov, I. & Hutter, F. (2019). "Decoupled Weight Decay Regularization." Why weight decay and L2 are different for Adam.
The fast.ai course: Practical Deep Learning for Coders, Lesson 11. Jeremy Howard builds the entire optimiser framework from scratch.
An overview of gradient descent optimisers: Ruder, S. (2016). "An overview of gradient descent optimization algorithms." Comprehensive survey of all major variants.
Next in this series: Batch normalisation from scratch, which stabilises training and allows higher learning rates.
Interactive Tools
- Gradient Descent Visualizer: Release all four optimisers from this post onto the same surface at once and watch them separate. The ravine preset is the momentum argument in one picture; Rosenbrock shows Adam finishing last at a shared learning rate
- 2D Convolution Visualizer: Visualise the forward pass that optimisers learn to tune, with element-wise multiplication and adjustable stride, padding, dilation
- Overfitting Explorer: See how model complexity and regularisation interact with the optimisation landscape
Related Posts
- Convolutional Neural Networks from Scratch: We benchmark our optimisers training this CNN on MNIST. The architecture is explained there.
- Backpropagation Demystified: Neural Nets from First Principles: Optimisers consume the gradients that backprop computes. This post shows how gradients flow through a network.
- Hyperparameter Optimisation: Grid vs Random vs Bayesian: Learning rate is the most important hyperparameter. This post covers systematic approaches to tuning it.
- Linear Regression from Scratch: Five Solvers, One Answer: Our SGD demonstration fits a linear regression, the same problem solved analytically in this post.
- Gaussian Process Regression from Scratch: Bayesian optimisation (covered in the hyperparameter post) uses GPs as the surrogate model.
- Perceptual Loss and Super-Resolution: Why Pixel MSE Fails: What you minimise matters as much as how: swapping pixel MSE for a feature-space objective changes what the optimiser converges to.
- The Universal Approximation Theorem in NumPy: Approximation power says a fit exists; optimisers are what actually reach it. The theorem defines what these algorithms are searching for.
- Hamiltonian Neural Networks: Momentum is itself a discretised physical system; Hamiltonian nets make that analogy the whole model.
- Kolmogorov-Arnold Networks in Python: A Curve on Every Edge: Adam training B-spline control points, and the loss spikes its normalised step causes near a very sharp minimum.









