Reinforcement Learning: From Q-Tables to Policy Gradients

12 tutorials· Ordered reading path

Reinforcement learning teaches agents to make decisions by trial and error. This reading path starts with the simplest tabular methods and builds toward deep RL and policy optimisation.

Each tutorial implements the algorithm from scratch in Python, so you can run, break, and rebuild every component. Start at the top and work down — each post assumes you have read the ones before it.

1

Q-Learning from Scratch: Navigating the Frozen Lake

Implement Q-learning from scratch on FrozenLake. The Bellman equation, temporal difference learning, epsilon-greedy exploration, and tabular RL in action.

2

Q-Learning for Games: Teaching an Agent Tic-Tac-Toe Through Self-Play

A Q-learning agent teaches itself tic-tac-toe through self-play. Tabular Q-learning in adversarial games, two agents bootstrapping toward optimal play.

3

Value Iteration vs Q-Learning: Dynamic Programming Meets RL

Two approaches to solving MDPs on FrozenLake: value iteration and Q-learning. The Bellman equation, DP sweeps, and why Q-learning needs no model.

4

Monte Carlo Tree Search in Python: Beating a Heuristic Without One

Build Monte Carlo tree search in Python and measure it against alpha-beta at the same node count. Four phases, one formula, no evaluation function, and it wins.

5

Reward Hacking in Reinforcement Learning

Watch a Q-learning agent game a naive reward: it farms a bonus tile forever and never reaches the goal. Then two fixes and a detector, from scratch in Python.

6

Deep Q-Networks: Experience Replay and Target Networks

Move beyond Q-tables to neural network function approximation. DQN from scratch in PyTorch with experience replay and target networks.

7

The Cross-Entropy Method: Solving RL Without Gradients

The Cross-Entropy Method: a derivative-free algorithm that solves CartPole by evolving a population of policies. No backpropagation, no value functions.

8

Solving CartPole Without Gradients: Simulated Annealing

Simulated annealing solves CartPole-v1 with 4 parameters and 800 episodes. The simplest derivative-free RL algorithm: perturb, evaluate, keep if better.

9

REINFORCE from Scratch in Python: Policy Gradients

The REINFORCE algorithm in pure NumPy on CartPole: forward pass, backprop, RMSProp, and why policy gradients optimise the policy instead of a value function.

10

A3C: Asynchronous Advantage Actor-Critic in PyTorch

Build A3C from scratch in PyTorch: a critic cuts REINFORCE's gradient variance, and asynchronous workers train it without a replay buffer, on CartPole.

11

Trust Region Methods: From REINFORCE to TRPO to PPO

REINFORCE, TRPO, and PPO from scratch in PyTorch on CartPole. Why vanilla policy gradients are unstable, how trust regions fix it, and why PPO won.

12

GRPO from Scratch: Group-Relative Advantage Without a Critic

Build GRPO from scratch in Python. The group-relative advantage that replaced PPO's critic in DeepSeek-R1, measured on a bandit, a maze and a verifier.

Related Tools

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Reinforcement Learning FAQ

Common questions about this topic