AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

MRPO: Magnitude-Regularized Policy Optimization via L1 Constraints

Wei Han, Yuanxing Liu, Mingda Li, Ruiyu Xiao, Weinan Zhang, Ting Liu

ICML 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace the KL constraint with an L1-norm constraint in GRPO-style trainingLlama3.1-8B-Instruct online preference alignment · AlpacaEval 2.0 WRwin rate (%): 53.97 -> 67.73Improved
replace the KL constraint with an L1-norm constraintQwen2.5-7B (RLVR) · Code-R1 pass@1pass@1 (%): 26.97 -> 33.56Improved
replace the KL constraint with an L1-norm constraintQwen2.5-7B (RLVR) · GSM8k pass@1pass@1 (%): 83.09 -> 84.08Improved
use L1 constraint instead of reverse KL with a weak 2B reward modelQwen2.5-7B-Instruct · AlpacaEval 2.0 WRwin rate (%): 39.62 -> 48.51Improved
use L1 constraint instead of forward KL or JS divergence (ablation)Llama3.1-8B-Instruct · AlpacaEval 2.0 WRwin rate (%): 50.17 -> 67.73 (DPH-JS -> MRPO-G)Improved
apply L1 constraint in offline RL with binary +1/-1 rewards instead of DPOGemma-7b-it · offline RL average over six benchmarksaverage score: 52.72 -> 53.20Improved
apply L1 constraint in offline RLGemma-7b-it · offline RL complex reasoningaverage score: 44.61 -> 45.40Improved
apply L1 constraint in offline RLGemma-7b-it · offline RL general capabilitiesaverage score: 60.82 -> 60.99No clear effect
use REINFORCE-based MRPO-R instead of GRPO-based MRPO-G (ablation)MRPO · Arena-Hard Gemini-2 (Llama3.1-8B)win rate (%): 81.82 -> 84.09 on Llama; 31.02 -> 29.68 on MistralMixed
use MRPO instead of GRPO on a small 1.5B modelQwen2.5-1.5B-Instruct · AlpacaEval 2.0 WRwin rate (%): 17.00 -> 28.46Improved
use MRPO instead of GRPO on a 3B modelQwen2.5-3B-Instruct · AlpacaEval 2.0 WRwin rate (%): 49.10 -> 62.25Improved
use MRPO on a base model with an already high win rate on AlpacaEval 1.0Llama3.1-8B-Instruct · AlpacaEval 1.0 WRwin rate (%): 97.34 -> 97.59Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy