Research map / Label-efficient and robust learning / Knowledge distillation

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLRT, which reverses the self-distillation signal on correct GRPO rollouts so tokens the student chose against the teacher are amplified as self-driven exploration.

Problem

Self-distillation in RLVR pulls the student toward a privileged-context teacher even on already-correct rollouts, overwriting the student's own successful reasoning and reducing exploration diversity.

Approach

Read the teacher-student log-ratio in reverse on correct rollouts: upweight tokens where the student's probability exceeds the teacher's (self-driven choices that still succeed), gated by reward, on top of GRPO. This is the inverse of RLSD's weight, with a reward gate.

Key results

• RLRT outperforms self-distillation baselines by an average of 8.9% on six math benchmarks, per the introduction.
• On Qwen3-4B-Base, RLRT reaches 22.5 avg@16 on AIME24 vs 15.0 for GRPO and 13.3 for RLSD.
• On Qwen3-8B, RLRT gives 70.6 avg@16 on AIME24 vs 70.2 for GRPO, with pass@16 of 93.3 vs 86.7.

Surprising findings

• SDPO collapses early on Qwen3 base models, with reward dropping to 0 within about 20 steps.
• SRPO, which routes correct rollouts to GRPO and incorrect ones to self-distillation, performs worse than full self-distillation on math.

Limitations

• Experiments restricted to mathematical reasoning.
• Relies on self-distillation where teacher and student share parameters.
• Gains on Qwen3-8B thinking model are small (about 2.2% average).

Experimental setup

Datasets: DAPO-Math-17k

Benchmarks: AIME24 AIME25 AIME26 HMMT26 AMC23 MATH500

Metrics: avg@16 pass@16 pass@k

Baselines: GRPO SDPO SRPO RLSD GRPO+EB DIVER

Builds on: GRPO RLSD SDPO OPSD DAPO

Closest papers by meaning

• RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning (AAAI 2026 · similarity 0.88)
• Reinforcement Learning Teachers of Test Time Scaling (NeurIPS 2025 · similarity 0.87)
• Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following (AAAI 2026 · similarity 0.87)
• Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.87)
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.87)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.86)