Research map / Reinforcement learning / Visual and representation RL
Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a short inference-time search (sampling, SGBS, online fine-tuning, or COMPASS latent search) over trained multi-agent RL policies breaks zero-shot performance ceilings on 17 hard RWARE, SMACv2 and Connector tasks, with up to 126% and average 45% gains.
Problem
Trained multi-agent RL policies, even when converged, plateau at a zero-shot performance ceiling on combinatorial, partially observable tasks, while many applications allow extra inference-time compute and wall-clock budget that is unused.
Approach
Formalises inference strategies under explicit time and parallel-compute budgets for Dec-POMDPs, adapts four paradigms (stochastic sampling, simulation-guided beam search, online fine-tuning, and COMPASS latent search with CMA-ES) to multi-agent settings, and benchmarks them across a grid of time and compute budgets.
Key results
- Inference-time search with a 30-second budget improves SABLE by more than 45% on aggregated performance across 17 tasks, with gains up to 126% on individual tasks.
- COMPASS is the leading inference strategy across tasks and base algorithms, and SABLE+COMPASS reaches close to 100% win rate where that metric is available.
- On Craftax-1B, 30-second stochastic sampling raises SABLE from 18.8 to 25.8 (about 37% gain).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stochastic sampling for 30 seconds instead of zero-shot inference | Sable · Craftax-1B | mean performance: 18.8 (18.1, 19.5) -> 25.8 (25.7, 26.1) | Helped |
| Add COMPASS inference-time search | Sable · 17-task multi-agent RL benchmark | aggregated performance: more than 45% | Helped |
| Use inference-time search instead of zero-shot execution | RL base policies · 17-task multi-agent RL benchmark | performance: up to a 126% and, on average, a 45% improvement | Helped |
| Use COMPASS rather than other inference strategies | Sable and other base policies · 17-task multi-agent RL benchmark | performance | Helped |
| Use online fine-tuning instead of stochastic sampling under a small time budget | RL base policies · 17-task multi-agent RL benchmark | performance | Hurt |
| Increase online fine-tuning compute capacity | Sable · Eight hardest multi-agent RL tasks | performance: 64 parallel attempts | Helped |
| Use online fine-tuning rather than COMPASS | Sable · Connector con-10x10x10a | performance | Helped |
| Use online fine-tuning rather than stochastic sampling | Sable · RWARE large-8ag | performance | Hurt |
| Use SGBS rather than other strategies in a low-budget regime | Sable · large-4ag and large-4ag-hard | performance: below 100 seconds and fewer than 10 parallel attempts | Helped |
| Use SGBS | Sable · Connector tasks | performance | Hurt |
| Increase compute capacity | Sable with COMPASS · Eight hardest multi-agent RL tasks | performance: over 95% win-rate when accessible | Helped |
| Increase compute capacity | Sable with stochastic sampling · Eight hardest multi-agent RL tasks | performance | Helped |
Surprising findings
- Under small time budgets, stochastic sampling outperforms online fine-tuning, which suffers from high variance and can reduce the number of attempts made.
- Zero-initialised COMPASS extra weights failed to create diversity, so random small initialisation was needed.
Limitations
- Focus is on multi-agent tasks in simulators with exact scores; real-world transfer and out-of-distribution behaviour are not evaluated.
- Online fine-tuning shows high variance and can be detrimental under tight budgets.
- Single-agent evidence is limited to one Craftax task and a 30-second budget.
Experimental setup
Datasets: RWARE SMACv2 Connector Craftax
Benchmarks: RWARE SMACv2 Connector Craftax
Metrics: normalised return win rate inter-quartile mean
Baselines: SABLE IPPO MAPPO Monte Carlo Tree Search VariBAD
Builds on: COMPASS simulation-guided beam search SABLE MAPPO IPPO
Closest papers by meaning
- Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference (ICLR 2025 · similarity 0.89)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.88)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.87)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.87)
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.87)
- TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.