Research map / Large language models / Reasoning and chain of thought

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that default on-policy self-distillation on math RL reduces to a conditional PMI signal that favors shortcut tokens, and proposes AntiSD, which reverses the sign by ascending JSD with an entropy-triggered gate, reaching GRPO accuracy in 2-10x fewer steps.

Problem

On-policy self-distillation for reasoning RL gives inconsistent gains on math, often underperforming GRPO, because the privileged-context teacher distorts the per-token credit signal.

Approach

Interprets the per-token self-distillation reward as conditional PMI between next token and privileged context, and reverses its sign by ascending Jensen-Shannon divergence (bounded advantage via softplus shape), gated off when teacher entropy collapses.

Key results

• AntiSD reaches GRPO's peak accuracy in 2-10x fewer steps across five models (e.g., 10x on Qwen3-4B-IT-2507).
• AntiSD improves final average accuracy over GRPO by +2.1 to +11.5 points; Qwen3-8B average 65.7 vs GRPO 57.4.
• Default self-distillation underperforms GRPO on every model (Qwen3-8B average 30.6 vs 57.4).

Surprising findings

• Default self-distillation collapses below the base model on several models, with Qwen3-8B average falling to 30.6 versus 57.4 for GRPO.
• Removing the teacher (self-reinforcement from student log-probs alone) collapses training within about 70 steps, indicating the gain depends on privileged information.

Limitations

• PMI characterization is local per-step, not a global-optimum statement for the combined objective.
• Gate threshold effects are model-conditional (0.90 threshold hurt Qwen3-4B by 8.3 points but not Qwen3-8B).
• Evaluation focuses on math reasoning with only a limited code probe; multi-turn agentic settings untested.

Experimental setup

Datasets: DAPO-Math-17k Dolci-RLZero code RL

Benchmarks: AIME 2024 AIME 2025 AIME 2026 HMMT 2025 MinervaMath HumanEval+ MBPP+

Metrics: avg@32 accuracy avg@4 accuracy speedup in steps to GRPO peak pass@k

Baselines: GRPO default self-distillation (SD) reverse-KL ascent No-teacher ablation No-gate ablation

Builds on: GRPO on-policy distillation self-distillation potential-based reward shaping verl

Closest papers by meaning

• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.89)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals (NeurIPS 2026 · similarity 0.89)
• Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.88)
• Rethinking On-Policy Self-Distillation for Thinking Models (NeurIPS 2026 · similarity 0.88)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.88)
• The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning (ICML 2026 · similarity 0.88)