Research map / Reinforcement learning / RL from feedback
Reinforcement Learning Teachers of Test Time Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains 7B Reinforcement-Learned Teachers (RLTs) that take question plus solution and write explanations, rewarded by student log-probabilities, so their raw outputs distill better students and cold-start RL than much larger teachers' postprocessed traces.
Problem
Standard RL for reasoning with one-hot correctness rewards suffers from exploration difficulty and is misaligned with the teacher's real role of producing distillation data; current distillation pipelines rely on large teachers and heuristic postprocessing.
Approach
Instead of solving problems from scratch, the teacher is given the question and ground-truth solution and asked to write step-by-step explanations; it is trained with dense rewards from a student's log-probabilities of the solution given the explanation, plus a KL term that keeps explanations interpretable as continuations of the question.
Key results
- RLT-7B (17K) reaches 49.50 overall (AIME 23.3, MATH500 82.8, GPQA 42.4), above Bespoke-7B at 46.60 and the 32B-scale baselines' overall scores except DeepSeek-R1.
- RLT-32B (17K) reaches 73.23 overall, above Bespoke-32B (71.47), which distills from DeepSeek-R1 traces.
- RLT-7B used to cold-start RL gives 50.53 overall versus 48.30 for Bespoke-7B + RL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill from raw RLT-7B traces rather than Bespoke-7B traces on the 17K set | Qwen2.5-7B student · GPQA Diamond | accuracy: 37.80 vs 42.40 | Helped |
| distill from raw RLT-7B traces rather than Bespoke-7B traces on the 1K subset | Qwen2.5-7B student · AIME 2024 | accuracy: 13.30 vs 20.00 | Helped |
| distill from RLT-7B traces rather than Bespoke-7B traces | Qwen2.5-32B student, 17K data · AIME 2024 | accuracy: 63.30 vs 66.70 | Helped |
| cold-start with RLT-7B traces rather than Bespoke-7B traces before RL | Qwen2.5-7B · AIME 2024 | accuracy: 16.70 vs 26.70 | Helped |
| cold-start with RLT-7B traces rather than Bespoke-7B traces before RL | Qwen2.5-7B · GPQA Diamond | accuracy: 45.40 vs 40.90 | Hurt |
| transfer RLT distillation zero-shot to countdown rather than use direct countdown RL | countdown student · Countdown | accuracy: 50.8 vs. 49.2 | Helped |
| train the RLT with RLOO instead of GRPO (ablation) | RLT-7B · Overall | accuracy: 49.50 vs 48.80 | No clear effect |
| use a 3B teacher instead of a 7B teacher (ablation) | RLT student distillation · Overall | accuracy: 49.50 vs 46.50 | Hurt |
| use a 3B teacher instead of a 7B teacher (ablation) | RLT-32B student distillation · Overall | accuracy: 73.20 vs 63.90 | Hurt |
Surprising findings
- Direct RL on the countdown task gives only marginal gains over an RL-free distillation baseline (50.8 vs 49.2), with over 98.5% overlap in solved problems.
- The highest-ranked traces from the 7B RLT before RL already yield about 90% of the performance gains of the R1 distillation baseline.
Limitations
- Reward and experiments rely on a Qwen2.5-7B student and a fixed question-solution corpus; the paper only begins to study the design space.
- Cost comparison with DeepSeek-R1 is a rough estimate and omits multiple-generation, filtering and postprocessing costs of baselines.
- Some baselines (e.g. Bespoke-32B and s1-32B) rely on reported results from prior work rather than re-run experiments.
Experimental setup
Datasets: Bespoke-Stratos-17k LIMR countdown (16K auto-generated) OpenThoughts-style SFT traces (open reasoning dataset)
Benchmarks: AIME 2024 MATH 500 GPQA Diamond LiveCodeBench OlympiadBench Countdown
Metrics: accuracy (single completion)
Baselines: Bespoke-7B Bespoke-32B s1-32B Sky-T1-32B QwQ-32B DeepSeek-R1 RL cold-start (GPT) + RL
Builds on: GRPO DeepSeek-R1 s1 Bespoke-Stratos
Closest papers by meaning
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.92)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.92)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
- AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning (AAAI 2026 · similarity 0.91)
- Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.