SGD, Momentum, and Adam: Optimisers from First Principles

Implement SGD, momentum, RMSProp, and Adam from scratch in NumPy. Visualise their trajectories on one loss surface, then benchmark them on a CNN on MNIST.

Cover Image for SGD, Momentum, and Adam: Optimisers from First Principles

Your network has calculated a gradient, but it still needs a rule for turning that information into a change to its weights. A step that is too large may overshoot, while small steps can make progress slow. The optimiser chooses those updates, and different choices can produce different training paths even with the same starting point.

We trace how stored gradients change each update by building SGD, momentum and Adam in NumPy. After 2,000 Rosenbrock steps, SGD remains 1.72 from the minimum while Adam is 0.0015 away. PyTorch checks and a CNN trained on MNIST test those implementations.

Let's Build It

First, let's watch all four optimisers race across the same loss surface. Click the badge to run this yourself:

Open In Colab

Animation showing SGD, SGD+Momentum, RMSProp, and Adam navigating the Rosenbrock function loss surface from the same starting point

Here is the complete code. We define all four optimisers and let them minimise the Rosenbrock function, a classic test surface with a narrow curved valley leading to the minimum at (1, 1).

import numpy as np

def rosenbrock(x, y):
    """Rosenbrock function: narrow curved valley, minimum at (1, 1)."""
    return (1 - x)**2 + 100 * (y - x**2)**2

def rosenbrock_grad(x, y):
    """Gradient of the Rosenbrock function."""
    dx = -2 * (1 - x) - 400 * x * (y - x**2)
    dy = 200 * (y - x**2)
    return np.array([dx, dy])

# --- SGD ---
def sgd(grad_fn, x0, lr=0.0002, steps=2000):
    path = [x0.copy()]
    x = x0.copy()
    for _ in range(steps):
        g = grad_fn(*x)
        x -= lr * g
        path.append(x.copy())
    return np.array(path)

# --- SGD + Momentum ---
def sgd_momentum(grad_fn, x0, lr=0.0002, mom=0.9, steps=2000):
    path = [x0.copy()]
    x = x0.copy()
    v = np.zeros_like(x)
    for _ in range(steps):
        g = grad_fn(*x)
        v = mom * v + g
        x -= lr * v
        path.append(x.copy())
    return np.array(path)

# --- RMSProp ---
def rmsprop(grad_fn, x0, lr=0.001, beta=0.99, eps=1e-8, steps=2000):
    path = [x0.copy()]
    x = x0.copy()
    v = np.zeros_like(x)
    for _ in range(steps):
        g = grad_fn(*x)
        v = beta * v + (1 - beta) * g**2
        x -= lr * g / (np.sqrt(v) + eps)
        path.append(x.copy())
    return np.array(path)

# --- Adam ---
def adam(grad_fn, x0, lr=0.005, beta1=0.9, beta2=0.999, eps=1e-8, steps=2000):
    path = [x0.copy()]
    x = x0.copy()
    m = np.zeros_like(x)  # first moment (mean of gradients)
    v = np.zeros_like(x)  # second moment (mean of squared gradients)
    for t in range(1, steps + 1):
        g = grad_fn(*x)
        m = beta1 * m + (1 - beta1) * g
        v = beta2 * v + (1 - beta2) * g**2
        m_hat = m / (1 - beta1**t)  # bias correction
        v_hat = v / (1 - beta2**t)  # bias correction
        x -= lr * m_hat / (np.sqrt(v_hat) + eps)
        path.append(x.copy())
    return np.array(path)

# Run all four from the same starting point
x0 = np.array([-1.0, 1.0])
path_sgd = sgd(rosenbrock_grad, x0, lr=0.0002, steps=2000)
path_mom = sgd_momentum(rosenbrock_grad, x0, lr=0.0002, mom=0.9, steps=2000)
path_rms = rmsprop(rosenbrock_grad, x0, lr=0.001, steps=2000)
path_adam = adam(rosenbrock_grad, x0, lr=0.005, steps=2000)

# Global minimum is at (1, 1)
for name, path in [("SGD", path_sgd), ("Momentum", path_mom),
                    ("RMSProp", path_rms), ("Adam", path_adam)]:
    dist = np.sqrt((path[-1][0] - 1)**2 + (path[-1][1] - 1)**2)
    print(f"{name:10s} final=({path[-1][0]:.4f}, {path[-1][1]:.4f}), dist to min={dist:.4f}")
SGD        final=(-0.5920, 0.3583), dist to min=1.7165
Momentum   final=( 0.9071, 0.8224), dist to min=0.2004
RMSProp    final=( 0.8133, 0.6607), dist to min=0.3872
Adam       final=( 0.9994, 0.9987), dist to min=0.0015

After 2,000 steps, vanilla SGD is still far from the minimum. Momentum reaches the valley and gets close. RMSProp adapts its learning rate per dimension but gets stuck in the curve. Adam combines both techniques and essentially solves it. Now let's understand why.

What Just Happened?

Vanilla SGD: The Simplest Optimiser

Stochastic gradient descent is the foundation of all neural network training. The update rule is one line:

θt+1=θt−η⋅∇L(θt)\theta_{t+1} = \theta_t - \eta \cdot \nabla L(\theta_t)

Where θ\theta are the parameters, η\eta is the learning rate, and ∇L\nabla L is the gradient of the loss. Compute the gradient, take a step in the opposite direction, repeat.

def sgd_step(params, grads, lr):
    """One step of vanilla SGD. This is the entire algorithm."""
    for p, g in zip(params, grads):
        p -= lr * g

The problem is that SGD treats every direction equally. If the loss surface is a narrow valley (steep walls, gentle slope along the floor), SGD oscillates across the walls while making slow progress along the floor. This is exactly what happens on the Beale function.

Loss contour plot showing SGD's oscillating path compared to momentum's smooth trajectory

Momentum: Remembering Past Gradients

Momentum fixes the oscillation problem by accumulating a running average of past gradients. Think of a ball rolling down a hill: it builds up speed in the consistent direction while the back-and-forth oscillations cancel out.

vt=β⋅vt−1+∇L(θt)v_t = \beta \cdot v_{t-1} + \nabla L(\theta_t)
θt+1=θt−η⋅vt\theta_{t+1} = \theta_t - \eta \cdot v_t

The velocity vv accumulates gradients over time. The momentum coefficient β\beta (typically 0.9) controls how much history to keep. With β=0.9\beta = 0.9, the effective window is roughly the last 10 gradients.

def sgd_momentum_step(params, grads, velocities, lr, mom=0.9):
    """SGD with momentum. Velocity accumulates past gradients."""
    for p, g, v in zip(params, grads, velocities):
        v[:] = mom * v + g
        p -= lr * v

This is the approach used in the fast.ai course, where Jeremy Howard implements it as a StatefulOptimizer with an AverageGrad statistic tracking the running average.

Dampened Momentum vs Classical Momentum

There are two conventions for momentum. The classical form (used above and in the fast.ai notebooks) computes vt=βvt−1+gtv_t = \beta v_{t-1} + g_t. The dampened form (used in Adam) computes vt=βvt−1+(1−β)gtv_t = \beta v_{t-1} + (1 - \beta) g_t. The dampening factor (1−β)(1 - \beta) keeps the running average at the same scale as the original gradients, which matters for bias correction.

# Classical momentum (fast.ai/PyTorch SGD default)
v = mom * v + grad

# Dampened momentum (used inside Adam)
v = mom * v + (1 - mom) * grad

With classical momentum, the running average is roughly ∑gi\sum g_i; with dampening, it is roughly mean(gi)\text{mean}(g_i). The dampened form is easier to reason about because vv stays on the same scale as the gradients regardless of β\beta.

Exponential Moving Average and Debiasing

Both momentum and Adam use exponential moving averages (EMAs). An EMA with coefficient β\beta gives exponentially decaying weight to older values:

EMAt=β⋅EMAt−1+(1−β)⋅xt\text{EMA}_t = \beta \cdot \text{EMA}_{t-1} + (1 - \beta) \cdot x_t

There is a problem at the start: EMA0=0\text{EMA}_0 = 0, so the first few estimates are biased towards zero. The fix is bias correction, which divides by (1−βt)(1 - \beta^t):

EMA^t=EMAt1−βt\widehat{\text{EMA}}_t = \frac{\text{EMA}_t}{1 - \beta^t}

This term starts near 0 when tt is small (strong correction) and approaches 1 as tt grows (no correction needed).

def ema_with_debiasing(values, beta=0.9):
    """Exponential moving average with bias correction."""
    avg = 0.0
    corrected = []
    for t, x in enumerate(values, 1):
        avg = beta * avg + (1 - beta) * x
        corrected.append(avg / (1 - beta**t))
    return corrected

Four panels showing momentum smoothing at beta=0.5, 0.7, 0.9, 0.99 with and without debiasing

The fast.ai notebook (dl2/09_optimizers.ipynb) has an excellent visualisation of this: with high β\beta, the undebiased average starts far too low and takes many steps to catch up. Debiasing fixes the initial estimates without affecting the long-run behaviour.

RMSProp: Adaptive Learning Rates

SGD and momentum use the same learning rate for every parameter. But some parameters might have consistently large gradients (they need smaller steps) while others have tiny gradients (they need larger steps). RMSProp adapts the learning rate per parameter by dividing by the root mean square of recent gradients.

vt=β⋅vt−1+(1−β)⋅gt2v_t = \beta \cdot v_{t-1} + (1 - \beta) \cdot g_t^2
θt+1=θt−ηvt+ϵ⋅gt\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t} + \epsilon} \cdot g_t

Parameters with large gradients get divided by a large number (smaller effective step). Parameters with small gradients get divided by a small number (larger effective step). This is why RMSProp reaches the minimum on the Beale function even though vanilla SGD cannot.

def rmsprop_step(params, grads, sq_avgs, lr, beta=0.9, eps=1e-8):
    """RMSProp: adapt learning rate by gradient magnitude."""
    for p, g, v in zip(params, grads, sq_avgs):
        v[:] = beta * v + (1 - beta) * g**2
        p -= lr * g / (np.sqrt(v) + eps)

RMSProp was proposed by Geoffrey Hinton in lecture 6e of his Coursera course (2012), never formally published in a paper. It remains widely used, especially in reinforcement learning.

Adam: The Best of Both Worlds

Adam (Adaptive Moment Estimation) combines momentum and RMSProp. It maintains two running averages: mtm_t (first moment, like momentum) and vtv_t (second moment, like RMSProp), both with bias correction.

mt=β1⋅mt−1+(1−β1)⋅gt(momentum)m_t = \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t \quad \text{(momentum)}
vt=β2⋅vt−1+(1−β2)⋅gt2(RMSProp)v_t = \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2 \quad \text{(RMSProp)}
m^t=mt1−β1t,v^t=vt1−β2t(bias correction)\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t} \quad \text{(bias correction)}
θt+1=θt−ηv^t+ϵ⋅m^t\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \cdot \hat{m}_t

Here is the complete implementation, matching the fast.ai notebook's adam_step:

class AdamNumPy:
    """Adam optimiser from scratch."""

    def __init__(self, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8):
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.eps = eps
        self.m = {}  # first moments
        self.v = {}  # second moments
        self.t = 0

    def step(self, params, grads):
        self.t += 1
        for i, (p, g) in enumerate(zip(params, grads)):
            if i not in self.m:
                self.m[i] = np.zeros_like(p)
                self.v[i] = np.zeros_like(p)

            self.m[i] = self.beta1 * self.m[i] + (1 - self.beta1) * g
            self.v[i] = self.beta2 * self.v[i] + (1 - self.beta2) * g**2

            m_hat = self.m[i] / (1 - self.beta1**self.t)
            v_hat = self.v[i] / (1 - self.beta2**self.t)

            p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)

Verifying Against PyTorch

Let's confirm our NumPy Adam produces the same parameter updates as torch.optim.Adam:

import torch

# Shared setup
np.random.seed(42)
w_np = np.array([1.0, -2.0, 0.5])
g_np = np.array([0.1, -0.3, 0.2])

# NumPy Adam
adam_np = AdamNumPy(lr=0.001)
w_test = w_np.copy()
adam_np.step([w_test], [g_np.copy()])

# PyTorch Adam
w_pt = torch.tensor(w_np, requires_grad=True)
opt = torch.optim.Adam([w_pt], lr=0.001)
w_pt.grad = torch.tensor(g_np)
opt.step()

print(f"NumPy:   {w_test}")
print(f"PyTorch: {w_pt.detach().numpy()}")
print(f"Match:   {np.allclose(w_test, w_pt.detach().numpy(), atol=1e-7)}")
NumPy:   [ 0.999 -1.999  0.499]
PyTorch: [ 0.999 -1.999  0.499]
Match:   True

Going Deeper

Weight Decay vs L2 Regularisation

The fast.ai notebook makes an important distinction: L2 regularisation adds wd⋅θ\text{wd} \cdot \theta to the gradient, while decoupled weight decay subtracts lr⋅wd⋅θ\text{lr} \cdot \text{wd} \cdot \theta directly from the parameters.

# L2 regularisation: modifies the gradient
grad += wd * param

# Decoupled weight decay: modifies the parameter directly
param *= (1 - lr * wd)

For vanilla SGD, these are mathematically equivalent. But for Adam, they are not. The AdamW paper (Loshchilov & Hutter, 2019) showed that decoupled weight decay generalises better with Adam because the adaptive learning rate doesn't interfere with the regularisation strength. PyTorch's torch.optim.AdamW implements this, and it is the default in most modern training recipes.

Diagram showing the difference between L2 regularisation (modifies gradient before Adam) and decoupled weight decay (applied directly to parameters after Adam step)

Training a CNN: SGD vs Adam in Practice

Let's benchmark our optimisers on a real task: training the SimpleCNN from our previous post on MNIST.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
train_data = datasets.MNIST('data', train=True, download=True, transform=transform)
test_data = datasets.MNIST('data', train=False, transform=transform)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=1000)

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 8, kernel_size=5, padding=2, stride=2)
        self.conv2 = nn.Conv2d(8, 16, kernel_size=3, padding=1, stride=2)
        self.conv3 = nn.Conv2d(16, 32, kernel_size=3, padding=1, stride=2)
        self.fc = nn.Linear(32 * 4 * 4, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = x.view(x.size(0), -1)
        return self.fc(x)

def train_and_evaluate(opt_class, opt_kwargs, epochs=5, seed=42):
    torch.manual_seed(seed)
    model = SimpleCNN()
    optimiser = opt_class(model.parameters(), **opt_kwargs)
    results = []

    for epoch in range(epochs):
        model.train()
        for images, labels in train_loader:
            loss = F.cross_entropy(model(images), labels)
            optimiser.zero_grad()
            loss.backward()
            optimiser.step()

        model.eval()
        correct = 0
        with torch.no_grad():
            for images, labels in test_loader:
                correct += (model(images).argmax(1) == labels).sum().item()
        results.append(correct / len(test_data) * 100)
    return results

configs = {
    "SGD (lr=0.01)":        (torch.optim.SGD, {"lr": 0.01}),
    "SGD+Momentum (lr=0.01)": (torch.optim.SGD, {"lr": 0.01, "momentum": 0.9}),
    "Adam (lr=0.001)":      (torch.optim.Adam, {"lr": 0.001}),
    "AdamW (lr=0.001)":     (torch.optim.AdamW, {"lr": 0.001, "weight_decay": 0.01}),
}

for name, (opt_cls, kwargs) in configs.items():
    acc = train_and_evaluate(opt_cls, kwargs)
    print(f"{name:25s} -> {[f'{a:.1f}%' for a in acc]}")
SGD (lr=0.01)             -> ['88.6%', '92.5%', '94.4%', '95.7%', '95.5%']
SGD+Momentum (lr=0.01)   -> ['97.3%', '98.1%', '98.4%', '98.5%', '98.5%']
Adam (lr=0.001)           -> ['96.7%', '97.7%', '98.2%', '98.6%', '98.4%']
AdamW (lr=0.001)          -> ['96.7%', '97.7%', '98.2%', '98.5%', '98.5%']

Training curves comparing SGD, SGD+Momentum, Adam, and AdamW on MNIST over 5 epochs

Vanilla SGD is dramatically slower, reaching only 95.5% after 5 epochs. Momentum transforms the picture: 97.3% after just one epoch, matching Adam's convergence speed. Adam and AdamW reach comparable final accuracy, with momentum matching AdamW at epoch 5 (98.5%) and edging past Adam (98.4%) thanks to its stronger gradient signal at this learning rate. The takeaway: momentum is the most impactful single improvement over vanilla SGD.

Choosing an Optimiser

OptimiserWhen to useTypical LR
SGDLarge-scale vision (with careful LR scheduling)0.01-0.1
SGD + MomentumDefault for CNNs when you want maximum generalisation0.01-0.1
AdamFast prototyping, NLP, GANs, any new project0.0001-0.001
AdamWProduction training with regularisation0.0001-0.001

A common observation in deep learning: Adam converges faster but SGD+momentum can generalise better if you invest time in learning rate scheduling. Many state-of-the-art vision models use SGD with cosine annealing. For most practitioners starting a new project, Adam is the right default.

When Optimisers Fail

Learning rate too high. The loss explodes or oscillates wildly. Adam is more forgiving than SGD because the adaptive denominator acts as an automatic brake, but even Adam diverges if the learning rate is absurdly high.

Learning rate too low. Training is stable but painfully slow. The loss decreases but never reaches a good minimum. This is especially common with vanilla SGD.

Gradient vanishing/exploding. If gradients are near zero (vanishing) or enormous (exploding), no optimiser can help. Fix the architecture first (batch normalisation, proper initialisation, skip connections).

Loss plateaus. The optimiser gets stuck on a flat region of the loss surface. Momentum helps because accumulated velocity carries it through flat spots. Learning rate warmup (starting low, then increasing) also helps by letting Adam's moment estimates stabilise before taking large steps.

Where This Comes From

The Robbins-Monro Framework (1951)

Stochastic gradient descent traces back to Herbert Robbins and Sutton Monro's 1951 paper, "A Stochastic Approximation Method." They proved that iterative updates using noisy gradient estimates converge to the optimum under mild conditions (the learning rate must decrease over time such that ∑ηt=∞\sum \eta_t = \infty and ∑ηt2<∞\sum \eta_t^2 < \infty).

"Given a function M(x) for which M(theta) = alpha, where alpha is known but M is unknown, we propose a method of estimating theta by successive approximations." -- Robbins & Monro (1951)

Modern SGD relaxes the decreasing learning rate requirement by using mini-batches (which reduce gradient noise) and learning rate schedules.

Momentum: Polyak (1964) and Nesterov (1983)

Boris Polyak introduced the heavy ball method in 1964, adding a momentum term to gradient descent. Yurii Nesterov improved it in 1983 with "accelerated gradient descent," which evaluates the gradient at the look-ahead position θ+βv\theta + \beta v rather than the current position. Nesterov momentum converges faster for convex problems, and PyTorch supports it via nesterov=True in torch.optim.SGD.

Adam: Kingma & Ba (2014)

Adam was introduced by Diederik Kingma and Jimmy Ba in their 2014 paper, "Adam: A Method for Stochastic Optimization". The key insight was combining the first moment (momentum) with the second moment (RMSProp) and adding bias correction for both.

"We propose Adam, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement. The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients." -- Kingma & Ba (2014)

Adam quickly became the default optimiser for deep learning. Its robustness to hyperparameter choices (the defaults β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8} work well across a wide range of problems) made it the practical choice for most researchers.

Beyond Adam

The fast.ai notebook implements LAMB (Layer-wise Adaptive Moments for Batch training), which scales Adam's update by the ratio of parameter norm to update norm. This enables training with very large batch sizes. Other recent variants include:

  • AdaFactor (Shazeer & Stern, 2018): Reduces memory by factorising the second moment matrix
  • LARS (You et al., 2017): Layer-wise adaptive rates for large-batch SGD
  • Lion (Chen et al., 2023): A simpler optimiser found by program search that uses only the sign of the momentum

Further Reading

Interactive Tools

  • Gradient Descent Visualizer: Release all four optimisers from this post onto the same surface at once and watch them separate. The ravine preset is the momentum argument in one picture; Rosenbrock shows Adam finishing last at a shared learning rate
  • 2D Convolution Visualizer: Visualise the forward pass that optimisers learn to tune, with element-wise multiplication and adjustable stride, padding, dilation
  • Overfitting Explorer: See how model complexity and regularisation interact with the optimisation landscape

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions