Research map / Reinforcement learning / RL from feedback
ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExPO, which generates positive training samples by having the model write self-explanations conditioned on the ground-truth answer, and plugs them into DPO and GRPO to unlock learning on hard math problems where the model initially fails.
Problem
GRPO-style RL post-training stalls on hard reasoning tasks where the base model produces no correct samples, so the advantage vanishes and learning signal is missing; expert CoTs are costly and often low-probability under the policy.
Approach
Generate positive samples as self-explanations conditioned on the ground-truth answer (c~ ~ pi(.|q,a*)), which are both in-distribution under the current policy and more likely to lead to the correct answer than the model's own CoTs; add a beta-weighted log-likelihood (ExP-SFT) term to GRPO and use online regenerated explanations for DPO.
Key results
- On MATH level-5, ExP-GRPO reaches 23% pass@4 versus 2% for GRPO and 12% for GRPO with SFT on expert ground-truth CoT (Qwen2.5-3B-Instruct).
- Online ExP-DPO improves MATH pass@4 from 54.3% (offline, explanation) to 60.4% and beats expert-CoT DPO (49.3%) on Qwen2.5-3B-Instruct.
- ExP-GRPO on the full MATH training set reaches 68.7% test accuracy with Qwen2.5-3B-Instruct versus 50.3% for GRPO and 61.9% for GRPO SFT-GT-CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online rather than offline self-explanations as ExP-DPO preferred responses | ExP-DPO · MATH | Pass@4: 45.7 -> 50.2; 54.3 -> 60.4 | Helped |
| Use online rather than offline self-explanations as ExP-DPO preferred responses | ExP-DPO · GSM8K | Pass@4: 71.2 -> 81.5; 80.1 -> 85.4 | Helped |
| Use self-generated explanations rather than expert CoT as the preferred response | Offline ExP-DPO · MATH and GSM8K | Pass@4: MATH: 45.7 vs 38.7; 54.3 vs 43.7; GSM8K: 71.2 vs 63.7; 80.1 vs 69.6; online LLaMA MATH: 50.2 vs 53.6 | Mixed |
| Add the ExP-SFT self-explanation guidance term (β=0.04) | GRPO · MATH level-4 | pass@4: 39% -> 76% | Helped |
| Add the ExP-SFT self-explanation guidance term (β=0.04) | GRPO · MATH level-5 | pass@4: 2% -> 23% | Helped |
| Use ExP-GRPO rather than GRPO SFT-GT-CoT | Qwen2.5-3B-Instruct · MATH level-4 and level-5 | pass@4: 65% -> 76%; 12% -> 23% | Helped |
| Remove the ExP-SFT term (β=0) (ablation) | ExP-GRPO · MATH training set (1 epoch) | accuracy: 68.7% -> 51.6%; 58.9% -> 44.3% | Hurt |
| Increase the guidance coefficient from β=0.04 to β=0.5 (ablation) | ExP-GRPO · MATH training set (1 epoch) | accuracy: 68.7% -> 47.3%; 58.9% -> 33.5% | Hurt |
| Reduce the guidance coefficient from β=0.04 to β=0.01 (ablation) | ExP-GRPO · MATH training set (1 epoch) | accuracy: 68.7% -> 60.4%; 58.9% -> 46.6% | Hurt |
| Set the guidance coefficient to β=0.1 or β=0.08 instead of β=0.04 (ablation) | ExP-GRPO · MATH training set (1 epoch) | accuracy: Qwen2.5: 68.7% vs 48.6%, 50.5%; LLaMA: 58.9% vs 34.8%, 35.3% | Hurt |
| Regenerate the positive explanation periodically from the current policy instead of keeping it fixed (ablation) | ExP-DPO · MATH and GSM8K | learning efficiency and final accuracy | Helped |
Surprising findings
- Expert-written CoTs as preferred responses in offline DPO yield deceptively low loss and only marginal downstream gains, sometimes underperforming self-generated explanations.
- Training with the expert CoT SFT term can underperform self-explanation guidance, indicating more expert-quality data does not automatically help.
Limitations
- Evaluated only on math reasoning (MATH, GSM8K) with 1-8B models; broader domains are discussed but not tested.
- Relies on a beta weight for the ExP-SFT term that must be tuned (beta=0.04 chosen by ablation); large beta degrades performance.
- Self-explanations may contain imperfect or hallucinated reasoning that could be reinforced.
Experimental setup
Datasets: MATH GSM8K
Benchmarks: MATH GSM8K
Metrics: accuracy pass@4 pass@64 pass@128
Baselines: GRPO GRPO SFT-GT-CoT offline DPO with expert CoT STaR
Builds on: GRPO DPO STaR
Closest papers by meaning
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.89)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.89)
- Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.