Research map / Large language models / Reasoning and chain of thought

Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies persistent high-loss "Rock Tokens" in on-policy distillation (Qwen3-30B-A3B teacher to Qwen3-4B student), shows they resist optimization and rarely matter causally, and proposes freezing their gradients to get a 1.4x wall-clock speedup.

Problem

On-policy distillation (OPD) uses per-token reverse KL, so high-loss tokens should shrink as the student aligns; the paper asks why a persistent set of high-loss tokens remains after training saturates and whether they matter for reasoning.

Approach

Defines Rock Tokens as token types whose high loss persists before and after OPD and recurs in similar contexts (Rock Score with context-consistency filter), then uses inference-time knockout and gradient-freezing of these tokens/windows to test whether they are functionally necessary.

Key results

• Rock Tokens account for a median of 18% of tokens in generated outputs while being about 6% of vocabulary types.
• Rock Tokens stay essentially unchanged in KL during OPD (median 0.21 to 0.19) while rare high-KL tokens drop about 58%.
• Only 3.5% (MATH-500) and 1.5% (IFEval) of screened Rock-Token candidates are Strong Pillars under knockout; freezing rock-token gradients gives a reported 1.4x wall-clock speedup.

Surprising findings

• Rock Tokens are stagnant in KL during OPD even though their gradients align with the global descent direction.
• No screened Rock Token candidate was a Strong Stumbling Block under inference-time knockout; all significant effects were negative.

Limitations

• Evaluation limited to competitive mathematics plus IFEval; open-ended generation and coding not tested.
• Single 30B-to-4B distillation pair; generality across architectures and scales unknown.
• Binary gradient-freeze mask is a blunt instrument; per-token pillar results are not significant after multiple-testing correction.

Experimental setup

Datasets: OpenThoughts3 MATH-500 AIME24 AIME25 HMMT25 IFEval

Benchmarks: AIME24 AIME25 HMMT25 MATH-500 IFEval

Metrics: Pass@1 accuracy delta (knockout)

Baselines: Original OPD Random freq-matched window freeze

Builds on: on-policy distillation KDFlow forking tokens (RLVR) TIP

Closest papers by meaning

• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.91)
• Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.90)
• Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.90)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
• SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.89)
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.88)