Research map / Reinforcement learning / RL from feedback
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
Zixuan Ren, Jinliang Lu, Junhong Wu, Yang Zhao, Dai Dai, Hua Wu, Haifeng Wang, Chengqing Zong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train task models with GRPO instead of SFT before merging | Llama-3.1-8B, TIEs merge · Five-task average | avg accuracy after merge: 49.9 -> 67.1 | Improved |
| train task models with GRPO instead of SFT before merging | Llama-3.1-8B, Averaging · Five-task average | avg accuracy after merge: 47.9 -> 60.1 | Improved |
| use PPO or Reinforce++ instead of GRPO (ablation) | RL merging · Math/Code/IF, TIEs | avg accuracy after merge: GRPO 72.0, PPO 69.3, Reinforce++ 68.4 | No clear effect |
| use RL instead of SFT on smaller/larger base model (ablation) | Llama-3.2-3B, Mistral-Small-24B · Math/Code/IF, TIEs | avg accuracy after merge: 3B: 35.1 -> 46.7; 24B: 70.4 -> 80.4 | Improved |
| use on-policy data (RL, RFT) instead of fixed SFT data (ablation) | training paradigm · Math update norm | parameter update norm: 6.5 -> 0.78 (RL), 2.36 (RFT) | Improved |
| remove negative-sample advantages from RL (RL-Pos) (ablation) | RL · Math/Code/IF | single-task accuracy and merge degradation: RL-Pos below RL but above SFT | Hurt |
| add noise perturbation in random direction (ablation) | SFT and RL models · performance landscape | task performance | No clear effect |
| increase number of merged models (TIEs) (ablation) | SFT vs RL · average performance | avg performance: SFT steeper decline than RL | Hurt |
| merge Puzzle-task models with TIEs (RL vs SFT) | GRPO · Puzzle | accuracy: SFT 35.8 vs RL 64.6; RL still drops 24% | Mixed |
Closest papers by meaning
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.92)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.90)
- Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR 2024 · similarity 0.90)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.89)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
