AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Knapsack RL: Compute-Efficient Reinforcement Learning via Heterogeneous Rollout Allocation

Ziniu Li, Congliang Chen, Tianyun Yang, Tian Ding, Ruoyu Sun, Ge Zhang, Wenhao Huang, Zhi-Quan Luo

ICML 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace homogeneous rollout allocation (N=8) with knapsack heterogeneous allocation at the same total budgetGRPO on DPSK-R1-Distill-1.5B · Six-benchmark averageavg@16: 45.9 -> 49.7Improved
replace homogeneous rollout allocation with knapsack allocationGRPO on Qwen3-4B-Base · Six-benchmark averageavg@16: 43.2 -> 45.1Improved
replace homogeneous rollout allocation with knapsack allocationGRPO on Qwen3-4B-Instruct · Six-benchmark averageavg@16: 59.2 -> 61.9Improved
replace homogeneous rollout allocation with knapsack allocationGRPO on Qwen2.5-Math-7B · Six-benchmark averageavg@16: 45.2 -> 47.5Improved
replace homogeneous rollout allocation with knapsack allocationGRPO on Qwen3-4B-Base · GPQA (out-of-domain)avg@16: 46.6 -> 45.5Hurt
replace homogeneous rollout allocation with knapsack allocationGRPO on Qwen3-4B-Instruct · AMCavg@16: 84.9 -> 83.1Hurt
replace homogeneous rollout allocation with knapsack allocation on low budget (ablation)GRPO on Qwen2.5-Math-7B · Rollout budget 1024 per iterationtest performance: 39.8 -> 45.5Improved
knapsack allocation at larger budgets (ablation)GRPO on Qwen2.5-Math-7B · Rollout budget 4096 per iterationtest performance: 47.8 -> 49.5Improved
knapsack allocation to raise effective-gradient ratioGRPO on Qwen2.5-Math-7B · DAPO-Math-17K trainingeffective-gradient ratio: roughly 40% increaseImproved
knapsack allocation reduces unsolved promptsGRPO on Qwen2.5-Math-7B · DAPO-Math-17K prompt status after trainingextremely-hard prompt count: 3,793 -> 3,596Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy