Research map / Large language models / Reasoning and chain of thought

RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLKD, an RL-based reasoning distillation framework where a generative structure reward model (GSRM) scores alignment between teacher and student meta-reasoning/solving step structures, combined with GRPO.

Problem

SFT-based reasoning distillation collapses the teacher's implicit multi-branch reasoning structure into a flat token sequence, so students imitate surface reasoning traces rather than learning authentic, generalizable reasoning.

Approach

Convert reasoning paths into sequences of meta-reasoning and solving (M, Q&A) steps with a trained generative reward model, then reward the student for step-wise structural alignment with the teacher's sequence, combining this structure reward with outcome accuracy in GRPO rather than SFT imitation.

Key results

• Qwen2.5-Math-7B-RLKD-Zero (RL only, 3.2K samples) reaches 23.3 pass@1 on AIME24 vs 16.7 for Qwen2.5-Math-7B-Instruct, and 34.9 vs 30.3 on GPQA-Diamond pass@1.
• DeepSeek-R1-Distill-Qwen-7B-RLKD improves AIME24 pass@1 (1 sample) from 50.0 to 53.3 and pass@16 from 80.0 to 86.7, while PPO and GRPO give little gain.
• On MATH-500 pass@1, DeepSeek-R1-Distill-Qwen-7B-RLKD scores 94.2 vs 92.4 for the base distilled model.

Surprising findings

• SFT on a domain-shifted subset progressively degrades performance while RLKD consistently improves it.
• Baseline RL methods (PPO, GRPO) bring little improvement over the SFT-distilled DeepSeek-R1-Distill-Qwen-7B.

Limitations

• GSRM relies on GPT-4o-generated annotations and a Qwen2.5-7B-based matcher, adding dependence on closed APIs and noisy judgments.
• Evaluation is limited to math and GPQA-Diamond; results on other reasoning domains are not reported.
• Several reported gains are small (e.g., 1-3 points on AIME24 pass@1) and based on few problems (30 AIME questions), so statistical reliability is limited.

Experimental setup

Datasets: OpenThoughts-114k OpenR1-Math OpenR1-Math-220k AIME24 MATH-500 GPQA-Diamond

Benchmarks: AIME24 MATH-500 GPQA-Diamond

Metrics: pass@1 pass@k

Baselines: Qwen2.5-Math-7B-Instruct DeepSeek-R1-Distill-Qwen-7B PPO GRPO SFT-based distillation

Builds on: GRPO PPO DAPO Open-R1 OpenThoughts-114k

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.92)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
• Reinforcement Learning Teachers of Test Time Scaling (NeurIPS 2025 · similarity 0.91)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.90)