Research map / Reinforcement learning / RL from feedback
Knapsack RL: Compute-Efficient Reinforcement Learning via Heterogeneous Rollout Allocation
Ziniu Li, Congliang Chen, Tianyun Yang, Tian Ding, Ruoyu Sun, Ge Zhang, Wenhao Huang, Zhi-Quan Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace homogeneous rollout allocation (N=8) with knapsack heterogeneous allocation at the same total budget | GRPO on DPSK-R1-Distill-1.5B · Six-benchmark average | avg@16: 45.9 -> 49.7 | Improved |
| replace homogeneous rollout allocation with knapsack allocation | GRPO on Qwen3-4B-Base · Six-benchmark average | avg@16: 43.2 -> 45.1 | Improved |
| replace homogeneous rollout allocation with knapsack allocation | GRPO on Qwen3-4B-Instruct · Six-benchmark average | avg@16: 59.2 -> 61.9 | Improved |
| replace homogeneous rollout allocation with knapsack allocation | GRPO on Qwen2.5-Math-7B · Six-benchmark average | avg@16: 45.2 -> 47.5 | Improved |
| replace homogeneous rollout allocation with knapsack allocation | GRPO on Qwen3-4B-Base · GPQA (out-of-domain) | avg@16: 46.6 -> 45.5 | Hurt |
| replace homogeneous rollout allocation with knapsack allocation | GRPO on Qwen3-4B-Instruct · AMC | avg@16: 84.9 -> 83.1 | Hurt |
| replace homogeneous rollout allocation with knapsack allocation on low budget (ablation) | GRPO on Qwen2.5-Math-7B · Rollout budget 1024 per iteration | test performance: 39.8 -> 45.5 | Improved |
| knapsack allocation at larger budgets (ablation) | GRPO on Qwen2.5-Math-7B · Rollout budget 4096 per iteration | test performance: 47.8 -> 49.5 | Improved |
| knapsack allocation to raise effective-gradient ratio | GRPO on Qwen2.5-Math-7B · DAPO-Math-17K training | effective-gradient ratio: roughly 40% increase | Improved |
| knapsack allocation reduces unsolved prompts | GRPO on Qwen2.5-Math-7B · DAPO-Math-17K prompt status after training | extremely-hard prompt count: 3,793 -> 3,596 | Improved |
Closest papers by meaning
- ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.90)
- AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.89)
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.89)
- DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.89)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
