Deep Learning Tutorials: 31 Implementations from Scratch

40 tutorials· Ordered reading path

Deep learning powers modern computer vision, NLP, and generative AI. This path builds your understanding from the ground up: you will implement each building block in NumPy, verify it matches PyTorch, then combine them into working systems.

Start with backpropagation to understand how gradients flow, then build convolutional networks for vision. From there, the path covers the training techniques (batch norm, optimisers, learning rate schedules) that make deep networks actually work in practice.

1

Backpropagation Demystified: Neural Nets from First Principles

Build a neural network from scratch and understand backpropagation through the chain rule. Gradient descent on XOR, tracing every weight update.

2

The Universal Approximation Theorem in NumPy

Why one hidden layer can fit any curve. Build a universal approximator in Python, sweep the width, watch the error collapse, and see why bumps tile anything.

3

Kolmogorov-Arnold Networks in Python: A Curve on Every Edge

Build a Kolmogorov-Arnold network in Python with NumPy. Learned B-spline functions on the edges, and edges that recover a formula's terms at R-squared 0.999999.

4

Convolutional Neural Networks from Scratch: Conv2d in NumPy

2D convolution from scratch in NumPy, verified against PyTorch's Conv2d, then a CNN trained on MNIST. Kernels, stride, padding, and feature maps explained.

5

Channel Attention in PyTorch: Five Seeds, No Effect

Squeeze-and-Excitation and ECA in eight lines of NumPy, then the ablation run properly. On a small CNN three seeds report a confident 0.24 point loss at 27 times its standard error, and five seeds report nothing at all.

6

Mechanistic Interpretability: One Neuron, Two Features

Mechanistic interpretability on a small MNIST CNN: activation maximisation, dataset examples and ablation disagree about the same neuron.

7

Graph Neural Networks from Scratch: Message Passing

Build a graph neural network from scratch in NumPy, then watch a vanilla GCN lose to a graph-blind MLP on a fraud graph, and learn the fixes that recover it.

8

SGD, Momentum, and Adam: Optimisers from First Principles

Implement SGD, momentum, RMSProp, and Adam from scratch in NumPy. Visualise their trajectories on one loss surface, then benchmark them on a CNN on MNIST.

9

Weight Initialization in PyTorch: Eight Layers, One Output

PyTorch's default weight initialization gives a plain ReLU network the same output for every input by eight layers deep. One line of He init fixes it.

10

Batch Normalisation from Scratch: Why It Works and When It Doesn't

Implement BatchNorm, LayerNorm, InstanceNorm, and GroupNorm from scratch in NumPy. See how each changes activation distributions, and learn when to use which.

11

Word2vec from Scratch: A Small Corpus, Weak Analogies

Build word2vec from scratch in NumPy, check its gradients against finite differences, and measure what one epoch on a small corpus can learn.

12

Collaborative Filtering from Scratch in Python

Build a movie recommender from scratch with embedding dot products. Implement EmbeddingDotBias in NumPy, train on MovieLens 100k, and read the embeddings.

13

Poincaré Embeddings in Python: Three Dimensions Beat Five Hundred

Poincaré embeddings in Python from scratch: three hyperbolic dimensions beat five hundred Euclidean ones on the WordNet mammals tree, reproducing Nickel and Kiela, with the rim costs measured.

14

Learning Rate Scheduling: From Step Decay to One-Cycle

Linear, cosine, exponential, and one-cycle learning rate schedules from scratch in PyTorch, with an LR finder, early stopping, and a one-cycle benchmark.

15

Backprop, Jacobians, Hessians: Why First-Order Wins

Why deep learning uses first-order optimisation: a PyTorch demo comparing SGD, L-BFGS, and Gauss-Newton on a GELU net, with Jacobians and Hessians.

16

LSTM from Scratch: The Forget Bias Sets the Memory

Build an LSTM from scratch, check the NumPy cell against PyTorch and measure how forget bias changes recall across 100 sequence steps.

17

Multi-Task Learning in PyTorch: When Gradients Conflict

Multi-task learning in PyTorch: measure gradient conflict between two tasks, implement PCGrad in ten lines, and find out when projecting gradients pays.

18

Bayesian Neural Networks: Laplace and Evidence Framework

Turn a point-estimate MLP into a Bayesian neural network with the Laplace approximation and the evidence framework. Fit a GELU net with error bars.

19

Transfer Learning for Text: Language Model to Sentiment

Fine-tune a pretrained language model on IMDb into a sentiment classifier: tokenisation, gradual unfreezing, and discriminative learning rates from scratch.

20

Perceptual Loss and Super-Resolution: Why Pixel MSE Fails

Build an image super-resolution model with perceptual (feature) loss from a pretrained VGG. Why pixel MSE gives blurry outputs and how deep features fix it.

21

Vision Transformers from Scratch in PyTorch

Build a Vision Transformer from scratch in PyTorch: patch embedding, class token, self-attention. Trained on CIFAR-10 and compared to a CNN.

22

A Transformer Without Training: 40 Digits, Exactly

Set every weight by hand in NumPy. A transformer without training adds 40-digit numbers exactly, while a trained model of the same shape fails past 13.

23

Variational Autoencoders in NumPy: Three Digits, No KL Term

Build a variational autoencoder from scratch in NumPy with hand-derived gradients. Measure what the KL term buys: without it the model can only generate 3 of 10 digits.

24

Diffusion Models from Scratch: Fifty Steps, If You Keep the Noise

Build a diffusion model from scratch in NumPy with hand-derived gradients. Measured over three seeds: DDIM's deterministic sampler is not worse on average, it is unreliable.

25

JEPA from Scratch: Predicting in Latent Space

Build a Joint-Embedding Predictive Architecture in PyTorch. Predict in latent space, watch representation collapse, and learn LeCun's world-model bet.

26

Next-Latent Prediction: Autoregression in Latent Space

Predict the next continuous latent instead of the next token: build autoregression in latent space from scratch and see why the head makes or breaks it.

27

Hamiltonian Neural Networks: Physics as Inductive Bias

Build a Hamiltonian Neural Network from scratch in PyTorch: learn a scalar energy, take its symplectic gradient, and conserve energy where a plain MLP drifts.

28

KV Cache Memory: Sizing LLM Inference

KV caching makes generation linear, not quadratic, but every cached token costs memory. The per-token sizing formula, the code, and what quantisation buys.

29

Shrinking the KV Cache: MQA, GQA, and Multi-Head Latent Attention

The KV cache is LLM serving's memory wall. MQA, GQA, and DeepSeek's Multi-Head Latent Attention shrink it 2 to 8x. We train all four and measure the cost.

30

Parallel LLM Decoding: Speculative and Diffusion

Autoregressive LLMs emit one token per pass. Build masked-diffusion and speculative decoders in NumPy to see how parallel decoding breaks the wall.

31

Beyond Attention: Linear Attention & Mamba SSMs

Build linear attention and a diagonal state-space model (Mamba) from scratch in PyTorch, and watch the O(n²) attention cost curve bend to O(n).

32

LLM Serving: Continuous Batching & PagedAttention

How vLLM serves hundreds of LLM requests at once: build continuous batching and a paged KV cache from scratch and watch GPU utilisation and throughput jump.

33

Mixture of Experts in LLMs: From Switch to DeepSeek-V3

Sparse MoE in PyTorch from scratch. Top-k routing, Switch Transformer load balancing, Mixtral, DeepSeek. Why MoE scales LLM parameters without scaling FLOPs.

34

Neural Network Quantization in Python: The Scale, Not the Bits

Quantize a trained network in NumPy and measure where accuracy goes. At 2 bits the scale rule is worth 80 accuracy points; five bits of range are worth 0.4.

35

LoRA from Scratch: Why LLM Fine-Tuning Got 1000x Cheaper

LoRA and QLoRA in pure PyTorch. Why low-rank adapters need 1000x fewer params, and how 4-bit base quantisation lets you fine-tune 65B models on one GPU.

36

TurboQuant: How a Random Rotation Makes LLM Quantization Near-Optimal

Google's TurboQuant from scratch in NumPy. Random rotations eliminate outlier-induced quantization errors for near-optimal LLM KV cache compression.

37

GRPO from Scratch: Group-Relative Advantage Without a Critic

Build GRPO from scratch in Python. The group-relative advantage that replaced PPO's critic in DeepSeek-R1, measured on a bandit, a maze and a verifier.

38

On-Policy Distillation: When Self-Generated Data Wins

On-policy distillation (GKD) vs off-policy SeqKD, from scratch in PyTorch. See exactly when training on a model's own output beats copying teacher transcripts.

39

Tabular Foundation Models: How TabPFN Predicts Without Training

TabPFN is a transformer pre-trained on millions of synthetic datasets that predicts in one forward pass: no training. Benchmarked against XGBoost.

40

Continual Learning: Training Neural Networks That Do Not Forget

Fine-tune a network on a new task and it erases the old. See catastrophic forgetting on split-MNIST, then fix it with Elastic Weight Consolidation and replay.

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Deep Learning FAQ

Common questions about this topic