Research map / Reinforcement learning / RL from feedback
Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Fuli Luo, Zhifang Sui
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Rollout Routing Replay (R3) | MoE inference-training KL · Qwen3-30B-A3B training vs inference | KL divergence: 1.5 × 10^-3 to 7.5 × 10^-4 | Improved |
| add R3 to GRPO | GRPO · Qwen3-30B-A3B-SFT mini_step=8 | average score: 48.84 -> 68.05 | Improved |
| replace GSPO with GRPO+R3 | GSPO · Qwen3-30B-A3B-SFT mini_step=8 | average score: 66.76 -> 68.05 | Improved |
| add R3 to GSPO | GSPO · Qwen3-30B-A3B-SFT mini_step=8 | average score: 66.76 -> 69.00 | Improved |
| replace TIS with R3 | GRPO+TIS · Qwen3-30B-A3B-SFT mini_step=1 | average score: 66.24 -> 71.83 | Improved |
| replace TIS with R3 | GRPO+TIS · Qwen3-30B-A3B-Base mini_step=1 | average score: 69.22 -> 70.73 | Improved |
| add TIS on top of R3 (ablation) | GRPO+R3 · Qwen3-30B-A3B-SFT mini_step=1 | average score: 71.83 -> 70.14 | Hurt |
| add TIS on top of R3 (ablation) | GRPO+R3 · Qwen3-30B-A3B-Base mini_step=1 | average score: 70.73 -> 69.17 | Hurt |
| add R3 to GRPO | GRPO · SWE-bench Verified | Pass@1: 31.80 -> 38.60 | Improved |
| add R3 to GRPO | GRPO · Qwen3-30B-A3B-ReasoningSFT mini_step=4 (vs GSPO) | average score: 74.62 -> 77.00 | Improved |
| add R3 to GRPO | GRPO · Qwen3-30B-A3B-ReasoningSFT mini_step=1 | average score: 75.30 -> 76.79 | Improved |
| add R3 to GRPO | GRPO · Qwen3-30B-A3B-Base mini_step=1 | average score: 61.69 -> 70.73 | Improved |
Closest papers by meaning
- Probing RLVR Training Instability through the Lens of Objective-Level Hacking (ICML 2026 · similarity 0.88)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.88)
- Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.87)
- STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.87)
- VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction (NeurIPS 2026 · similarity 0.87)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
