40 tutorials· Ordered reading path
Deep learning powers modern computer vision, NLP, and generative AI. This path builds your understanding from the ground up: you will implement each building block in NumPy, verify it matches PyTorch, then combine them into working systems.
Start with backpropagation to understand how gradients flow, then build convolutional networks for vision. From there, the path covers the training techniques (batch norm, optimisers, learning rate schedules) that make deep networks actually work in practice.
Build a neural network from scratch and understand backpropagation through the chain rule. Gradient descent on XOR, tracing every weight update.
Why one hidden layer can fit any curve. Build a universal approximator in Python, sweep the width, watch the error collapse, and see why bumps tile anything.
Build a Kolmogorov-Arnold network in Python with NumPy. Learned B-spline functions on the edges, and edges that recover a formula's terms at R-squared 0.999999.
2D convolution from scratch in NumPy, verified against PyTorch's Conv2d, then a CNN trained on MNIST. Kernels, stride, padding, and feature maps explained.
Squeeze-and-Excitation and ECA in eight lines of NumPy, then the ablation run properly. On a small CNN three seeds report a confident 0.24 point loss at 27 times its standard error, and five seeds report nothing at all.
Mechanistic interpretability on a small MNIST CNN: activation maximisation, dataset examples and ablation disagree about the same neuron.
Build a graph neural network from scratch in NumPy, then watch a vanilla GCN lose to a graph-blind MLP on a fraud graph, and learn the fixes that recover it.
Implement SGD, momentum, RMSProp, and Adam from scratch in NumPy. Visualise their trajectories on one loss surface, then benchmark them on a CNN on MNIST.
PyTorch's default weight initialization gives a plain ReLU network the same output for every input by eight layers deep. One line of He init fixes it.
Implement BatchNorm, LayerNorm, InstanceNorm, and GroupNorm from scratch in NumPy. See how each changes activation distributions, and learn when to use which.
Build word2vec from scratch in NumPy, check its gradients against finite differences, and measure what one epoch on a small corpus can learn.
Build a movie recommender from scratch with embedding dot products. Implement EmbeddingDotBias in NumPy, train on MovieLens 100k, and read the embeddings.
Poincaré embeddings in Python from scratch: three hyperbolic dimensions beat five hundred Euclidean ones on the WordNet mammals tree, reproducing Nickel and Kiela, with the rim costs measured.
Linear, cosine, exponential, and one-cycle learning rate schedules from scratch in PyTorch, with an LR finder, early stopping, and a one-cycle benchmark.
Why deep learning uses first-order optimisation: a PyTorch demo comparing SGD, L-BFGS, and Gauss-Newton on a GELU net, with Jacobians and Hessians.
Build an LSTM from scratch, check the NumPy cell against PyTorch and measure how forget bias changes recall across 100 sequence steps.
Multi-task learning in PyTorch: measure gradient conflict between two tasks, implement PCGrad in ten lines, and find out when projecting gradients pays.
Turn a point-estimate MLP into a Bayesian neural network with the Laplace approximation and the evidence framework. Fit a GELU net with error bars.
Fine-tune a pretrained language model on IMDb into a sentiment classifier: tokenisation, gradual unfreezing, and discriminative learning rates from scratch.
Build an image super-resolution model with perceptual (feature) loss from a pretrained VGG. Why pixel MSE gives blurry outputs and how deep features fix it.
Build a Vision Transformer from scratch in PyTorch: patch embedding, class token, self-attention. Trained on CIFAR-10 and compared to a CNN.
Set every weight by hand in NumPy. A transformer without training adds 40-digit numbers exactly, while a trained model of the same shape fails past 13.
Build a variational autoencoder from scratch in NumPy with hand-derived gradients. Measure what the KL term buys: without it the model can only generate 3 of 10 digits.
Build a diffusion model from scratch in NumPy with hand-derived gradients. Measured over three seeds: DDIM's deterministic sampler is not worse on average, it is unreliable.
Build a Joint-Embedding Predictive Architecture in PyTorch. Predict in latent space, watch representation collapse, and learn LeCun's world-model bet.
Predict the next continuous latent instead of the next token: build autoregression in latent space from scratch and see why the head makes or breaks it.
Build a Hamiltonian Neural Network from scratch in PyTorch: learn a scalar energy, take its symplectic gradient, and conserve energy where a plain MLP drifts.
KV caching makes generation linear, not quadratic, but every cached token costs memory. The per-token sizing formula, the code, and what quantisation buys.
The KV cache is LLM serving's memory wall. MQA, GQA, and DeepSeek's Multi-Head Latent Attention shrink it 2 to 8x. We train all four and measure the cost.
Autoregressive LLMs emit one token per pass. Build masked-diffusion and speculative decoders in NumPy to see how parallel decoding breaks the wall.
Build linear attention and a diagonal state-space model (Mamba) from scratch in PyTorch, and watch the O(n²) attention cost curve bend to O(n).
How vLLM serves hundreds of LLM requests at once: build continuous batching and a paged KV cache from scratch and watch GPU utilisation and throughput jump.
Sparse MoE in PyTorch from scratch. Top-k routing, Switch Transformer load balancing, Mixtral, DeepSeek. Why MoE scales LLM parameters without scaling FLOPs.
Quantize a trained network in NumPy and measure where accuracy goes. At 2 bits the scale rule is worth 80 accuracy points; five bits of range are worth 0.4.
LoRA and QLoRA in pure PyTorch. Why low-rank adapters need 1000x fewer params, and how 4-bit base quantisation lets you fine-tune 65B models on one GPU.
Google's TurboQuant from scratch in NumPy. Random rotations eliminate outlier-induced quantization errors for near-optimal LLM KV cache compression.
Build GRPO from scratch in Python. The group-relative advantage that replaced PPO's critic in DeepSeek-R1, measured on a bandit, a maze and a verifier.
On-policy distillation (GKD) vs off-policy SeqKD, from scratch in PyTorch. See exactly when training on a model's own output beats copying teacher transcripts.
TabPFN is a transformer pre-trained on millions of synthetic datasets that predicts in one forward pass: no training. Benchmarked against XGBoost.
Fine-tune a network on a new task and it erases the old. See catastrophic forgetting on split-MNIST, then fix it with Elastic Weight Consolidation and replay.
Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.
Common questions about this topic