A smaller model learns by copying a larger model's answers, then makes a mistake when it has to generate an answer on its own. The next choice now depends on a history that may never have appeared in the training examples. On-policy distillation lets the student practise from its own attempts, with the teacher providing guidance along the way. That makes the source of the training data an important part of the comparison.
Using student histories changes the training cost. Thinking Machines Lab reported comparable reasoning performance at a fraction of RL's cost in October 2025, extending Agarwal and colleagues and Gu and colleagues. We compare teacher-generated data with student rollouts on a small task to test when that exposure helps.
Off-policy (SeqKD) distillation learns from teacher-generated sequences, while on-policy (GKD) learns from student rollouts. On our small, fully supervised bracket task, the off-policy student learns better from its cleaner gradients. We build both in about 40 lines of PyTorch and test how the source of training sequences changes the result.
A task where one wrong token derails everything
To see exposure bias we need a task where a single bad token is unrecoverable. Addition will not do: predict a wrong digit and you are still on a perfectly valid state the teacher handles fine. We want the opposite, a task where one slip throws you off a cliff.
Balanced brackets are exactly that. A valid string in the Dyck-1 language opens and closes brackets so the running depth never goes negative and ends at zero: (()()) is valid, ())( is not. Emit one extra ) at depth zero and the string is dead, no continuation can save it. The "reasoning" here is minimal but real: track the nesting depth, and never close what you have not opened.
Our teacher is an oracle. Given any prefix we know the current depth and the steps remaining , and we can count exactly how many valid completions go through ( versus ) using the ballot numbers. That gives a precise per-token target distribution for any prefix, including the broken ones a student invents. No teacher training, so the only moving part is the student.
import torch, torch.nn as nn, torch.nn.functional as F
OPEN, CLOSE, BOS, VOCAB = 0, 1, 2, 3
L = 12 # bracket pairs; sequences are 2L symbols long
T = 2 * L
# Ncomp[d][r] = number of length-r continuations that stay >= 0 and end at depth 0
Ncomp = [[0] * (T + 1) for _ in range(T + 2)]
for d in range(T + 1):
Ncomp[d][0] = 1 if d == 0 else 0
for r in range(1, T + 1):
for d in range(T + 1):
opn = Ncomp[d + 1][r - 1] if d + 1 <= T else 0
cls = Ncomp[d - 1][r - 1] if d - 1 >= 0 else 0
Ncomp[d][r] = opn + cls
def oracle_probs(d, r):
"""P(open), P(close) for a valid completion from depth d with r steps left."""
tot = Ncomp[d][r] if 0 <= d <= T and r > 0 else 0
if tot: # a valid completion exists
p_open = (Ncomp[d + 1][r - 1] if d + 1 <= T else 0) / tot
return p_open, 1 - p_open
return (0.0, 1.0) if d > 0 else (1.0, 0.0) # off-manifold: close to recover
The oracle is the whole point of the experiment. It is a perfect teacher that knows the right move from every state, even states no valid string ever visits. That lets us isolate the one thing we care about: not how good the teacher is, but which states the student gets trained on.
Two ways to distil the same teacher
Both methods minimise the same forward KL divergence to the same oracle. They differ in one thing only: which sequences the student trains on.
Off-policy (SeqKD). Train on ground-truth valid strings, teacher-forced. At every position the student matches the oracle's distribution, but it only ever conditions on correct prefixes. It never sees a broken state.
On-policy (GKD). The student generates its own string by sampling, the oracle scores each of those self-generated states, and the student matches the oracle on its own trajectory. The training prefixes are whatever the student actually produces, broken ones included.

def distill_step(student, seq, mode, opt, oracle_targets):
if mode == "on": # student rolls out its own sequence
with torch.no_grad():
roll = seq[:, :1].clone() # start from BOS
for _ in range(T):
p = F.softmax(student(roll)[:, -1, :2], -1)
nxt = torch.multinomial(p, 1)
roll = torch.cat([roll, nxt], 1)
seq = roll
target = oracle_targets(seq) # oracle distribution at every position
logp = F.log_softmax(student(seq)[:, :T], -1)
loss = F.kl_div(logp, target, reduction="batchmean")
opt.zero_grad(); loss.backward()
nn.utils.clip_grad_norm_(student.parameters(), 1.0); opt.step()
return loss.item()
The student is a deliberately small Transformer (a single layer, , about 3,800 parameters) so it has to make trade-offs. We warm both branches with a few epochs of off-policy training so the student is competent enough that its rollouts are not pure noise, then split: one branch keeps going off-policy, the other switches to on-policy. After training we measure validity, the fraction of the student's own free-running samples that are balanced strings.

Both methods work. Both drive the tiny student from random scribbling to over 98% valid brackets. If you were hoping for a dramatic gap, there is not one: the two finish in a dead heat, and off-policy is the smoother, faster learner on the way there. The only visible difference is the dip where on-policy switches to its own rollouts and has to climb back. That is not a bug in the code. It is the first half of the lesson.
What just happened
Off-policy distillation had everything it needed. The oracle supervises every state, the training strings are drawn from the same distribution the student generates at test time, and the student is small enough that clean teacher-forced gradients beat the noisier signal from its own rollouts. When the training distribution already covers the states the student visits, there is nothing for on-policy to fix.
That is the subtlety the headlines skip. On-policy distillation is not a free accuracy boost. It pays off only when off-policy data fails to cover the states the student actually reaches, and on a matched, fully-supervised toy that gap does not exist.
So we have to look for the gap somewhere off the happy path. The greedy student stays valid almost always, but what happens when it is pushed off its comfortable trajectory, the way a real model is when it samples at temperature or makes a genuine mistake halfway through a long answer?
@torch.no_grad()
def validity_at_temperature(model, temp, n=3000):
seq = torch.full((n, 1), BOS, device=DEVICE)
depth = torch.zeros(n, device=DEVICE)
alive = torch.ones(n, dtype=torch.bool, device=DEVICE)
for _ in range(T):
logits = model(seq)[:, -1, :2] / temp # hotter = more deviation
tok = torch.multinomial(F.softmax(logits, -1), 1).squeeze(1)
alive &= ~((tok == CLOSE) & (depth == 0)) # illegal close kills the string
depth = (depth + torch.where(tok == OPEN, 1, -1)).clamp(min=0)
seq = torch.cat([seq, tok[:, None]], 1)
return (alive & (depth == 0)).float().mean().item()

Even here, pushed off the greedy path, the off-policy student holds up at least as well as the on-policy one. On this toy, with a teacher that covers every state and a train/test distribution that already matches, on-policy distillation has no edge to exploit. The honest experimental result is a tie, and it tells you precisely what on-policy needs in order to win.
When on-policy distillation actually wins
The mechanism is distribution coverage, and it has a name in the imitation-learning literature: exposure bias, or covariate shift. Write the off-policy objective as the forward KL between teacher and student, averaged over the teacher's data distribution:
The student is supervised on , the states the teacher visits. At inference it is evaluated on , the states it visits. When those two distributions match, off-policy is fine. When they diverge, the student is undersupervised exactly where it needs help most. On-policy distillation changes the expectation to the student's own distribution:
Now the training distribution is the deployment distribution by construction. There is no gap to fall into. GKD generalises this with a mixing coefficient that interpolates between the two, so is pure off-policy and is pure on-policy, and it lets you swap the forward KL for reverse KL or a Jensen-Shannon divergence.
Three conditions decide whether that change of distribution is worth its cost.
A real coverage gap. The teacher's transcripts must miss states the student reaches. On our bracket toy the oracle covers everything, so there is no gap. On a long chain-of-thought, the teacher's clean reasoning never contains the student's own wrong turns, so the gap is enormous and grows with every additional step.
A student large enough to optimise on its own rollouts. Our 3,800-parameter student learns more cleanly from teacher forcing than from its own noisy samples. A multi-billion-parameter model is stable enough that on-policy training converges, which is why the effect shows up at LLM scale and not at toy scale.
Mode-seeking where it matters. When the student cannot represent the teacher exactly, forward KL spreads its probability mass over teacher modes it cannot reach, producing bland or invalid samples. MiniLLM's reverse KL and on-policy sampling both push the student to commit to the modes it can represent, which is why Thinking Machines and MiniLLM both report on-policy methods beating sequence-level KD on generation quality.
None of those conditions hold on a small, clean, fully-supervised task. All of them hold when you are distilling a reasoning model. That is the whole story: on-policy distillation is a scale and distribution-shift technique, not a universal upgrade.
Practical notes and failure modes
On-policy training is less stable than off-policy. In our runs the on-policy student would occasionally spike to near-perfect validity and then degrade if training continued, the same overtraining collapse you see in reinforcement learning. Keep the best checkpoint by held-out score rather than the last one, and stop early.
It is also more expensive. Every gradient step needs a fresh student rollout plus a teacher forward pass to score it, so a step costs roughly a generation plus a teacher evaluation, against off-policy's single teacher-free forward pass on cached data. The KV-cache is what makes the rollout affordable in practice.
Do not reach for on-policy distillation when a cheaper option suffices. If your teacher's transcripts already cover your deployment distribution, if your student is small, or if plain sequence-level KD already hits your target, the extra machinery buys you nothing. Reach for it when the student must generate long outputs whose error states the teacher never demonstrates: multi-step reasoning, agentic tool use, code.
The foundations
Knowledge distillation starts with Hinton, Vinyals and Dean (2015), "Distilling the Knowledge in a Neural Network." Their insight was that a teacher's full output distribution carries more information than the hard label. The "dark knowledge" in the runner-up probabilities, softened by a temperature , tells the student which wrong answers are plausible:
For classifiers, the teacher's data is the student's data, so off-policy is the natural and only choice. The problem appears for autoregressive models, where the student generates one token at a time and conditions on its own past outputs.
Agarwal et al. (2024) named and fixed it. Their Generalized Knowledge Distillation introduces the mixture between teacher data and student-generated data, and the freedom to choose the divergence:
We propose GKD, which addresses [the distribution mismatch] by training the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences.
Setting recovers pure on-policy distillation; recovers classical sequence-level KD. MiniLLM (Gu et al., 2024) arrived at the same place from the divergence side, arguing that reverse KL is the right objective for a capacity-limited student because it is mode-seeking rather than mode-covering.
The 2025 Thinking Machines Lab post is the modern recipe: a student rollout, dense per-token teacher grading, reverse KL, applied to maths reasoning and chat assistants. Their framing is the cleanest one-liner for the whole idea, that on-policy distillation "unites the error-correcting relevance of RL with the reward density of SFT." Our bracket experiment is the negative control that shows why that union only pays off once the student is large enough and the task long enough for the train-inference gap to open up. The mechanism is the same one behind on-policy reinforcement learning, which we cover in Trust Region Methods: from REINFORCE to TRPO to PPO.
Interactive Tools
- LLM Cost Calculator: Distillation exists to make a capable model cheaper to serve; price out teacher versus student inference here
- Q-Learning Visualiser: The on-policy versus off-policy distinction comes from reinforcement learning; watch it play out in a gridworld
Related Posts
- LoRA from Scratch: Why LLM Fine-Tuning Got 1000x Cheaper: Another way to specialise a model cheaply, often combined with distillation
- Mixture of Experts in LLMs: From Switch to DeepSeek-V3: Sparse capacity as an alternative route to efficient large models
- KV Cache Memory: Sizing LLM Inference: The KV-cache that makes on-policy student rollouts affordable
- Trust Region Methods: From REINFORCE to TRPO to PPO: Where on-policy versus off-policy learning comes from
- Transfer Learning for Text: Language Model to Sentiment: Moving knowledge between models the supervised way
- Continual Learning: Training Neural Networks That Do Not Forget: Retaining a model's capabilities across successive training stages, the flip side of distilling new behaviour into it
- The AI Jobpocalypse, Your Kids, and Chicken Impossible: A wider take on where capable, cheaper-to-run models like these distilled students are heading
- Parallel LLM Decoding: Speculative and Diffusion: Distillation shrinks the student; parallel decoding speeds up the rollouts that on-policy training depends on, attacking the same inference-cost wall from the latency side
- LLM Serving: Continuous Batching and PagedAttention: Distillation shrinks the model you serve; this post covers how the serving stack then keeps it busy and cheap









