Research map / Reinforcement learning / RL from feedback
Reinforcement Learning via Self-Distillation
Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, Andreas Krause
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GRPO scalar advantages with SDPO self-teacher advantages | RLVR post-training · LiveCodeBench v6, Qwen3-8B | accuracy (%): 41.2 -> 48.8 | Improved |
| use SDPO instead of GRPO with scalar rewards only | Qwen3-8B and Olmo3-7B-Instruct · science Q&A and tool use (aggregate) | final accuracy (%): 64.1 -> 68.8 | Improved |
| use SDPO instead of GRPO for generation length | science Q&A, Olmo3-7B-Instruct Chemistry · Chemistry | response length: 7x reduction | Improved |
| use logit-level instead of token-level or sequence-level SDPO credit assignment (ablation) | SDPO · LiveCodeBench v6, Qwen3-8B | accuracy | Improved |
| sequence-level SDPO (rich feedback without dense credit) versus GRPO (ablation) | SDPO · LiveCodeBench v6, Qwen3-8B | accuracy | Improved |
| scale model size from small to large Qwen3 (ablation) | SDPO versus GRPO · LiveCodeBench v6, Qwen3 0.6B to 8B | accuracy | Improved |
| use SDPO on the weakest model Qwen2.5-1.5B (ablation) | SDPO · LiveCodeBench v6, Qwen2.5-1.5B | accuracy | Hurt |
| add environment output to own-solution feedback for the self-teacher (ablation) | SDPO · LiveCodeBench v6, Qwen3-8B | student accuracy (%): 42.6 -> 48.3 | Improved |
| include the student's original attempt in the teacher prompt (ablation) | SDPO · LiveCodeBench v6, Qwen3-8B | student accuracy (%): 48.3 -> 44.5 | Hurt |
| use a regularized teacher (trust-region or EMA) instead of unregularized current teacher (ablation) | SDPO · LiveCodeBench v6, Qwen3-8B | best accuracy (%): 36.1 -> 50.6 (trust-region) | Improved |
| update the teacher via trust-region or EMA instead of freezing it (ablation) | SDPO · LiveCodeBench v6, Qwen3-8B | best accuracy (%): 48.8 -> 50.6 (trust-region) | Improved |
| add GRPO advantages to SDPO (lambda 0.9) (ablation) | SDPO · LiveCodeBench v6, Qwen3 sizes | accuracy | Mixed |
Closest papers by meaning
- Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.90)
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.90)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
- Procedural Memory Distillation: Online Reflection for Self-Improving Language Models (NeurIPS 2026 · similarity 0.90)
- From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.90)
- SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
