Research map / Reinforcement learning / RL from feedback
Mirage or Method? How Model–Task Alignment Induces Divergent RL Conclusions
Haoze Wu, Cheng Wang, Wenshuo Zhao, Junxian He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ground-truth reward with random reward (ablation) | DAPO · MATH500 | accuracy (%): 71.0 -> 57.5 (still above base 40.8) | Mixed |
| replace ground-truth reward with random reward (ablation) | DAPO Qwen2.5-7B · SynLogic (weak alignment) | accuracy (%): 42.6 -> 10.2 | Hurt |
| replace ground-truth reward with incorrect reward (ablation) | DAPO Llama-3.1-8B-Instruct · MATH500 (weak alignment) | accuracy (%): 32.5 -> 26.4 | Hurt |
| replace ground-truth reward with majority-vote reward (ablation) | DAPO Qwen2.5-7B · SynLogic | accuracy (%): 1.5 -> 2.8 | No clear effect |
| train with majority-vote reward (ablation) | DAPO Qwen2.5-7B · MATH500 | accuracy (%): 40.8 -> 69.4 | Improved |
| TTRL on strongly aligned Qwen2.5-7B | Qwen2.5-7B · MATH500 | accuracy (%): 40.8 -> 62.1 | Improved |
| TTRL on weakly aligned task | Qwen2.5-7B · SynLogic | accuracy (%): 1.5 -> 1.8 | No clear effect |
| one-shot RL instead of full dataset, strong alignment (ablation) | DAPO Qwen2.5-7B · MATH500 | accuracy (%): 71.0 -> 65.2 (base 40.8) | Mixed |
| one-shot RL on weakly aligned logic tasks (ablation) | DAPO Qwen2.5-7B · SynLogic | accuracy (%): 1.5 -> 0.8 | Hurt |
| variance-based sample selection instead of random single sample (ablation) | one-shot RL Qwen2.5-7B · MATH500 | accuracy (%): selected 65.2 vs random 58.7 and 63.0 | No clear effect |
| negative-sample-only training (NSR) instead of full DAPO, strong alignment (ablation) | DAPO Qwen2.5-7B · MATH500 | accuracy (%): 71.0 -> 68.7 | No clear effect |
| negative-sample-only training (NSR) in weak alignment (ablation) | DAPO Qwen2.5-7B · SynLogic | accuracy (%): 42.6 -> 1.5 | Hurt |
Closest papers by meaning
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.90)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.89)
- The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
