On-Policy Distillation: When Self-Generated Data Wins

On-policy distillation (GKD) vs off-policy SeqKD, from scratch in PyTorch. See exactly when training on a model's own output beats copying teacher transcripts.

Cover Image for On-Policy Distillation: When Self-Generated Data Wins

A smaller model learns by copying a larger model's answers, then makes a mistake when it has to generate an answer on its own. The next choice now depends on a history that may never have appeared in the training examples. On-policy distillation lets the student practise from its own attempts, with the teacher providing guidance along the way. That makes the source of the training data an important part of the comparison.

Using student histories changes the training cost. Thinking Machines Lab reported comparable reasoning performance at a fraction of RL's cost in October 2025, extending Agarwal and colleagues and Gu and colleagues. We compare teacher-generated data with student rollouts on a small task to test when that exposure helps.

Off-policy (SeqKD) distillation learns from teacher-generated sequences, while on-policy (GKD) learns from student rollouts. On our small, fully supervised bracket task, the off-policy student learns better from its cleaner gradients. We build both in about 40 lines of PyTorch and test how the source of training sequences changes the result.

A task where one wrong token derails everything

To see exposure bias we need a task where a single bad token is unrecoverable. Addition will not do: predict a wrong digit and you are still on a perfectly valid state the teacher handles fine. We want the opposite, a task where one slip throws you off a cliff.

Balanced brackets are exactly that. A valid string in the Dyck-1 language opens and closes brackets so the running depth never goes negative and ends at zero: (()()) is valid, ())( is not. Emit one extra ) at depth zero and the string is dead, no continuation can save it. The "reasoning" here is minimal but real: track the nesting depth, and never close what you have not opened.

Our teacher is an oracle. Given any prefix we know the current depth dd and the steps remaining rr, and we can count exactly how many valid completions go through ( versus ) using the ballot numbers. That gives a precise per-token target distribution for any prefix, including the broken ones a student invents. No teacher training, so the only moving part is the student.

import torch, torch.nn as nn, torch.nn.functional as F

OPEN, CLOSE, BOS, VOCAB = 0, 1, 2, 3
L = 12            # bracket pairs; sequences are 2L symbols long
T = 2 * L

# Ncomp[d][r] = number of length-r continuations that stay >= 0 and end at depth 0
Ncomp = [[0] * (T + 1) for _ in range(T + 2)]
for d in range(T + 1):
    Ncomp[d][0] = 1 if d == 0 else 0
for r in range(1, T + 1):
    for d in range(T + 1):
        opn = Ncomp[d + 1][r - 1] if d + 1 <= T else 0
        cls = Ncomp[d - 1][r - 1] if d - 1 >= 0 else 0
        Ncomp[d][r] = opn + cls

def oracle_probs(d, r):
    """P(open), P(close) for a valid completion from depth d with r steps left."""
    tot = Ncomp[d][r] if 0 <= d <= T and r > 0 else 0
    if tot:                                   # a valid completion exists
        p_open = (Ncomp[d + 1][r - 1] if d + 1 <= T else 0) / tot
        return p_open, 1 - p_open
    return (0.0, 1.0) if d > 0 else (1.0, 0.0)  # off-manifold: close to recover

The oracle is the whole point of the experiment. It is a perfect teacher that knows the right move from every state, even states no valid string ever visits. That lets us isolate the one thing we care about: not how good the teacher is, but which states the student gets trained on.

Open In Colab

Two ways to distil the same teacher

Both methods minimise the same forward KL divergence to the same oracle. They differ in one thing only: which sequences the student trains on.

Off-policy (SeqKD). Train on ground-truth valid strings, teacher-forced. At every position the student matches the oracle's distribution, but it only ever conditions on correct prefixes. It never sees a broken state.

On-policy (GKD). The student generates its own string by sampling, the oracle scores each of those self-generated states, and the student matches the oracle on its own trajectory. The training prefixes are whatever the student actually produces, broken ones included.

Animated training curves: after a shared off-policy warmup, the off-policy and on-policy students both climb to high bracket validity, finishing neck and neck.

def distill_step(student, seq, mode, opt, oracle_targets):
    if mode == "on":                       # student rolls out its own sequence
        with torch.no_grad():
            roll = seq[:, :1].clone()      # start from BOS
            for _ in range(T):
                p = F.softmax(student(roll)[:, -1, :2], -1)
                nxt = torch.multinomial(p, 1)
                roll = torch.cat([roll, nxt], 1)
            seq = roll
    target = oracle_targets(seq)           # oracle distribution at every position
    logp = F.log_softmax(student(seq)[:, :T], -1)
    loss = F.kl_div(logp, target, reduction="batchmean")
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(student.parameters(), 1.0); opt.step()
    return loss.item()

The student is a deliberately small Transformer (a single layer, dmodel=16d_{model}=16, about 3,800 parameters) so it has to make trade-offs. We warm both branches with a few epochs of off-policy training so the student is competent enough that its rollouts are not pure noise, then split: one branch keeps going off-policy, the other switches to on-policy. After training we measure validity, the fraction of the student's own free-running samples that are balanced strings.

Both off-policy and on-policy distillation drive the small student to around 98% bracket validity; on-policy dips when it switches to its own rollouts after the shared warmup, then catches up.

Both methods work. Both drive the tiny student from random scribbling to over 98% valid brackets. If you were hoping for a dramatic gap, there is not one: the two finish in a dead heat, and off-policy is the smoother, faster learner on the way there. The only visible difference is the dip where on-policy switches to its own rollouts and has to climb back. That is not a bug in the code. It is the first half of the lesson.

What just happened

Off-policy distillation had everything it needed. The oracle supervises every state, the training strings are drawn from the same distribution the student generates at test time, and the student is small enough that clean teacher-forced gradients beat the noisier signal from its own rollouts. When the training distribution already covers the states the student visits, there is nothing for on-policy to fix.

That is the subtlety the headlines skip. On-policy distillation is not a free accuracy boost. It pays off only when off-policy data fails to cover the states the student actually reaches, and on a matched, fully-supervised toy that gap does not exist.

So we have to look for the gap somewhere off the happy path. The greedy student stays valid almost always, but what happens when it is pushed off its comfortable trajectory, the way a real model is when it samples at temperature or makes a genuine mistake halfway through a long answer?

@torch.no_grad()
def validity_at_temperature(model, temp, n=3000):
    seq = torch.full((n, 1), BOS, device=DEVICE)
    depth = torch.zeros(n, device=DEVICE)
    alive = torch.ones(n, dtype=torch.bool, device=DEVICE)
    for _ in range(T):
        logits = model(seq)[:, -1, :2] / temp        # hotter = more deviation
        tok = torch.multinomial(F.softmax(logits, -1), 1).squeeze(1)
        alive &= ~((tok == CLOSE) & (depth == 0))     # illegal close kills the string
        depth = (depth + torch.where(tok == OPEN, 1, -1)).clamp(min=0)
        seq = torch.cat([seq, tok[:, None]], 1)
    return (alive & (depth == 0)).float().mean().item()

Validity falls as sampling temperature rises for both students; the off-policy student degrades no faster than the on-policy one on this matched-distribution toy.

Even here, pushed off the greedy path, the off-policy student holds up at least as well as the on-policy one. On this toy, with a teacher that covers every state and a train/test distribution that already matches, on-policy distillation has no edge to exploit. The honest experimental result is a tie, and it tells you precisely what on-policy needs in order to win.

When on-policy distillation actually wins

The mechanism is distribution coverage, and it has a name in the imitation-learning literature: exposure bias, or covariate shift. Write the off-policy objective as the forward KL between teacher and student, averaged over the teacher's data distribution:

Loff=Ex∼pteacher[DKL(pteacher(⋅∣x) ∥ pstudent(⋅∣x))]\mathcal{L}_{\text{off}} = \mathbb{E}_{x \sim p_{\text{teacher}}} \big[ D_{KL}(p_{\text{teacher}}(\cdot \mid x) \,\|\, p_{\text{student}}(\cdot \mid x)) \big]

The student is supervised on x∼pteacherx \sim p_{\text{teacher}}, the states the teacher visits. At inference it is evaluated on x∼pstudentx \sim p_{\text{student}}, the states it visits. When those two distributions match, off-policy is fine. When they diverge, the student is undersupervised exactly where it needs help most. On-policy distillation changes the expectation to the student's own distribution:

Lon=Ex∼pstudent[DKL(pteacher(⋅∣x) ∥ pstudent(⋅∣x))]\mathcal{L}_{\text{on}} = \mathbb{E}_{x \sim p_{\text{student}}} \big[ D_{KL}(p_{\text{teacher}}(\cdot \mid x) \,\|\, p_{\text{student}}(\cdot \mid x)) \big]

Now the training distribution is the deployment distribution by construction. There is no gap to fall into. GKD generalises this with a mixing coefficient λ\lambda that interpolates between the two, so λ=0\lambda = 0 is pure off-policy and λ=1\lambda = 1 is pure on-policy, and it lets you swap the forward KL for reverse KL or a Jensen-Shannon divergence.

Three conditions decide whether that change of distribution is worth its cost.

A real coverage gap. The teacher's transcripts must miss states the student reaches. On our bracket toy the oracle covers everything, so there is no gap. On a long chain-of-thought, the teacher's clean reasoning never contains the student's own wrong turns, so the gap is enormous and grows with every additional step.

A student large enough to optimise on its own rollouts. Our 3,800-parameter student learns more cleanly from teacher forcing than from its own noisy samples. A multi-billion-parameter model is stable enough that on-policy training converges, which is why the effect shows up at LLM scale and not at toy scale.

Mode-seeking where it matters. When the student cannot represent the teacher exactly, forward KL spreads its probability mass over teacher modes it cannot reach, producing bland or invalid samples. MiniLLM's reverse KL and on-policy sampling both push the student to commit to the modes it can represent, which is why Thinking Machines and MiniLLM both report on-policy methods beating sequence-level KD on generation quality.

None of those conditions hold on a small, clean, fully-supervised task. All of them hold when you are distilling a reasoning model. That is the whole story: on-policy distillation is a scale and distribution-shift technique, not a universal upgrade.

Practical notes and failure modes

On-policy training is less stable than off-policy. In our runs the on-policy student would occasionally spike to near-perfect validity and then degrade if training continued, the same overtraining collapse you see in reinforcement learning. Keep the best checkpoint by held-out score rather than the last one, and stop early.

It is also more expensive. Every gradient step needs a fresh student rollout plus a teacher forward pass to score it, so a step costs roughly a generation plus a teacher evaluation, against off-policy's single teacher-free forward pass on cached data. The KV-cache is what makes the rollout affordable in practice.

Do not reach for on-policy distillation when a cheaper option suffices. If your teacher's transcripts already cover your deployment distribution, if your student is small, or if plain sequence-level KD already hits your target, the extra machinery buys you nothing. Reach for it when the student must generate long outputs whose error states the teacher never demonstrates: multi-step reasoning, agentic tool use, code.

The foundations

Knowledge distillation starts with Hinton, Vinyals and Dean (2015), "Distilling the Knowledge in a Neural Network." Their insight was that a teacher's full output distribution carries more information than the hard label. The "dark knowledge" in the runner-up probabilities, softened by a temperature τ\tau, tells the student which wrong answers are plausible:

LKD=τ2 DKL ⁣(softmax(zteacher/τ) ∥ softmax(zstudent/τ))\mathcal{L}_{KD} = \tau^2 \, D_{KL}\!\left( \text{softmax}(z_{\text{teacher}} / \tau) \,\|\, \text{softmax}(z_{\text{student}} / \tau) \right)

For classifiers, the teacher's data is the student's data, so off-policy is the natural and only choice. The problem appears for autoregressive models, where the student generates one token at a time and conditions on its own past outputs.

Agarwal et al. (2024) named and fixed it. Their Generalized Knowledge Distillation introduces the λ\lambda mixture between teacher data and student-generated data, and the freedom to choose the divergence:

We propose GKD, which addresses [the distribution mismatch] by training the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences.

Setting λ=1\lambda = 1 recovers pure on-policy distillation; λ=0\lambda = 0 recovers classical sequence-level KD. MiniLLM (Gu et al., 2024) arrived at the same place from the divergence side, arguing that reverse KL is the right objective for a capacity-limited student because it is mode-seeking rather than mode-covering.

The 2025 Thinking Machines Lab post is the modern recipe: a student rollout, dense per-token teacher grading, reverse KL, applied to maths reasoning and chat assistants. Their framing is the cleanest one-liner for the whole idea, that on-policy distillation "unites the error-correcting relevance of RL with the reward density of SFT." Our bracket experiment is the negative control that shows why that union only pays off once the student is large enough and the task long enough for the train-inference gap to open up. The mechanism is the same one behind on-policy reinforcement learning, which we cover in Trust Region Methods: from REINFORCE to TRPO to PPO.

Interactive Tools

  • LLM Cost Calculator: Distillation exists to make a capable model cheaper to serve; price out teacher versus student inference here
  • Q-Learning Visualiser: The on-policy versus off-policy distinction comes from reinforcement learning; watch it play out in a gridworld

Stay in the loop

Get notified when new tutorials and deep-dives are published. No spam, unsubscribe anytime.

Related Posts

Frequently Asked Questions