Research map / Reinforcement learning / RL from feedback
Approximation of Log-Partition Function in Policy Mirror Descent Induces Implicit Regularization for LLM Post-Training
Zhenghao Xu, Qin Lu, Changlong Yu, Tuo Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GRPO with PMD-MEAN under staleness 16 | Qwen2.5-7B · AIME 2025 | Avg@32: 10.52 -> 19.48 | Improved |
| replace GRPO with PMD-MEAN under staleness 16 | Qwen2.5-7B · AIME 2024 | Avg@32: 17.08 -> 19.69 | Improved |
| replace GRPO with PMD-MEAN (tau 0.1) under staleness 16 | Qwen3-30B-A3B-Base · AIME 2024 | Avg@32: 36.56 -> 50.83 | Improved |
| use larger global rollout batch (stale rollouts) instead of on-policy gradient | PMD-MEAN · Qwen2.5-7B | overall time (ms/token): 0.0569 -> 0.0126 ms/token | Improved |
| compare PMD-MEAN to on-policy gradient at matched quality | PMD-MEAN · Qwen2.5-7B | Avg@32: 18.49 -> 19.58 average | No clear effect |
| compare PMD-MEAN to GSPO | PMD-MEAN · Qwen3-30B-A3B-Base | Avg@32: 43.96 -> 44.01 average | No clear effect |
| compare PMD-MEAN to GSPO | PMD-MEAN · Qwen2.5-7B | Avg@32: 13.75 -> 19.58 average | Improved |
| fit partition-normalized target (PMD-PART) instead of mean-reward approximation (ablation) | PMD-MEAN · Qwen2.5-7B training stability | training stability: unstable, may collapse | Hurt |
| vary tau of PMD-MEAN (ablation) | PMD-MEAN · Qwen2.5-7B | Avg@32: average 19.58, 17.55, 19.58 for tau 0.005, 0.01, 0.02 | Mixed |
Closest papers by meaning
- Policy Mirror Descent with Lookahead (NeurIPS 2024 · similarity 0.91)
- Learning mirror maps in policy mirror descent (ICLR 2025 · similarity 0.90)
- Operator World Models for Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity (NeurIPS 2026 · similarity 0.89)
- Sharp Convergence and Sample Complexity of Policy Mirror Descent for Average-Reward MDPs (NeurIPS 2026 · similarity 0.89)
- Convergence of Policy Mirror Descent Beyond Compatible Function Approximation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
