Research map / Reinforcement learning / RL from feedback
On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement
Wenlong Deng, Yushu Li, Boying Gong, Yi Ren, Christos Thrampoulidis, Xiaoxiao Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LLDS likelihood-preserving regularizer | GRPO, Qwen2.5-3B-Base · 7 QA benchmarks avg | EM: 0.312 -> 0.360 | Improved |
| add LLDS-MA (answer-token masking) (ablation) | GRPO, Qwen2.5-3B-Base · 7 QA benchmarks avg | EM: 0.360 -> 0.440 (+45.2% over GRPO) | Improved |
| add LLDS | GRPO, Qwen2.5-3B-Instruct · 7 QA benchmarks avg | EM: 0.336 -> 0.441 | Improved |
| add LLDS | GRPO, Qwen2.5-7B-Base · 7 QA benchmarks avg | EM: 0.350 -> 0.480 | Improved |
| add LLDS | GRPO, Qwen2.5-7B-Instruct · 7 QA benchmarks avg | EM: 0.396 -> 0.483 | Improved |
| add LLDS to GSPO | GSPO, Qwen2.5-3B-Instruct · 7 QA benchmarks avg | EM: 0.331 -> 0.451 | Improved |
| regularization weight lambda 0.01 instead of 0 (ablation) | GRPO + LLDS, Qwen2.5-3B-Instruct · training reward | collapse step: collapse ~200 -> ~220 | Mixed |
| regularization weight lambda 0.2 instead of 0 (ablation) | GRPO + LLDS, Qwen2.5-3B-Instruct · training reward | training stability | Improved |
| use LLDS instead of dense-reward Tree-GRPO | GRPO, Qwen2.5-7B · 7 QA benchmarks avg | EM: 0.439 -> 0.480 | Improved |
| add LLDS across model sizes and variants | GRPO · training reward curves | collapse | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Tree-GRPO — TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models (ICLR 2026) · better on 3 of 3 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.93)
- Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.91)
- Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents (ICML 2026 · similarity 0.91)
- ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning (ICLR 2026 · similarity 0.90)
- Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
