Research map / Reinforcement learning / Visual and representation RL
Position: Benchmarking is Limited in Reinforcement Learning Research
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that rigorous RL benchmarking is computationally prohibitive (needs ~100+ seeds per algorithm-environment pair) and that scientific testing should be a primary complementary experimentation paradigm, illustrated with exploration experiments on a four-rooms MDP.
Problem
Rigorous performance benchmarking in RL requires many seeds and environments, making it computationally prohibitive and statistically unreliable with few trials, and it reveals only which algorithm works, not why.
Approach
Argues that benchmarking should not be the sole evidence for algorithmic claims; proposes "scientific testing" (controlled experiments aimed at understanding how an algorithm works) as a primary complementary paradigm, and quantifies benchmarking's seed requirements via bootstrap CI simulations on classic control and gridworld tasks.
Key results
- Bootstrap 95% confidence intervals only achieve the target failure rate (0.05) with at least about 100 seeds per algorithm-environment pair under uniform weighting, so few-seed benchmarks cannot be trusted.
- At 100 seeds, average CI width is about 0.183 under adversarial weighting versus 0.150 under uniform weighting.
- A survey of 144 non-theory NeurIPS 2022 RL papers found 91% included benchmarking but only 35% included scientific testing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adversarial rather than uniform weighting | RL benchmarking with bootstrap confidence intervals · 100 seeds per algorithm-environment pair | 95% confidence interval width: approximately 0.150 -> approximately 0.183 | Hurt |
| Use uniform weighting rather than adversarial weighting | RL benchmarking with bootstrap confidence intervals · algorithm-environment comparisons | minimum seeds for reliable confidence intervals: greater than 100 | Mixed |
| Increase the number of seeds per algorithm-environment pair | RL benchmarking with bootstrap confidence intervals · algorithm-environment comparisons | confidence interval reliability: at least 10 (Adversarial-Both), 50 (Uniform-Both) or 100 (Adversarial-Env) | Helped |
| Add a restart distribution to Sarsa(λ) | Sarsa(λ) exploration · Four rooms MDP | policy improvement across the state space | Helped |
| Sample start states uniformly over the state space | Sarsa(λ) exploration · Four rooms MDP | policy improvement across the state space | Helped |
| Add intrinsic motivation to Sarsa(λ) | Sarsa(λ) exploration · Four rooms MDP | state-action entropy | Mixed |
| Set intrinsic reward magnitude β to 5.0 (ablation) | Intrinsic Motivation and restart distribution + Intrinsic Motivation · Four rooms MDP | extrinsic reward learning: β=5.0 | Hurt |
| Set intrinsic reward magnitude β too low (ablation) | Intrinsic Motivation · Four rooms MDP | exploration | Hurt |
| Add restart distribution to Intrinsic Motivation | Sarsa(λ) exploration · Four rooms MDP, far room | time spent in far-room states | Helped |
| Use restart-based exploration rather than intrinsic motivation alone (ablation) | Sarsa(λ) exploration · Four rooms MDP, far room | time spent in far-room states | Helped |
| Replace bootstrap confidence intervals with t-distribution-based intervals (ablation) | RL benchmarking · aggregate performance across environments | confidence interval width | Hurt |
Surprising findings
- Bootstrap intervals fail badly at small sample sizes, so common practice of 1, 3, or 5 seeds yields unreliable comparisons.
- Restart-based exploration methods did not converge close to the optimal value function even after many episodes.
Limitations
- Experiments limited to classic small control and discrete MDPs with tabular or simple linear Sarsa(λ) agents, and a single bootstrap method.
- Bootstrap intervals are not guaranteed valid; the paper uses them as an approximate lower bound on required samples.
- Scientific testing does not guarantee easier statistical analysis and its benefits are demonstrated only in a toy setting.
Experimental setup
Datasets: Cart-Pole Mountain Car Acrobot Pinball chain MDPs gridworlds four rooms MDP
Benchmarks: Cart-Pole Mountain Car Acrobot Pinball chain MDPs gridworlds four rooms MDP
Metrics: confidence interval width confidence interval failure rate (coverage) number of algorithms statistically tied with the best return distance to epsilon-greedy optimal value function state-action entropy
Baselines: Sarsa(λ) Q(λ) Actor-Critic with eligibility traces NAC-TD PPO
Builds on: rliable performance profiles count-based intrinsic motivation restart-based exploration
Closest papers by meaning
- Benchmarking Quantum Reinforcement Learning (ICML 2025 · similarity 0.88)
- A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.87)
- Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (ICML 2024 · similarity 0.87)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.87)
- Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.