Research map / Reinforcement learning / Visual and representation RL
Empirical Design in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a practitioner-oriented guide to empirical design in RL, proposing tolerance intervals, idealized-performance estimation, and controlled randomness, supported by DQN/SAC/DDPG case-study experiments on small classic-control environments.
Problem
Empirical RL results are often unreliable: untuned baselines, short episode cutoffs, skewed/bimodal performance distributions, and confidence intervals that miss the mean make conclusions about algorithm utility misleading.
Approach
Combines statistical tools (tolerance intervals, paired testing with controlled seeds, idealized-performance estimation that avoids maximization bias) with a framework treating hyperparameters as unknowns and a cross-validation-like view of hyperparameter optimization across environment sets, illustrated on simple RL testbeds.
Key results
- Roughly 96% of 1000 simulated DQN hyperparameter-sweep experiments on Mountain Car over-report the average performance of the best configuration.
- DQN with default hyperparameters fails on Lunar Lander, while a small amount of tuning brings it to very good performance.
- Episode cutoffs noticeably change algorithm performance, indicating short cutoffs make environments easier for agents.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use short episode cutoffs instead of longer cutoffs | RL agents in the studied environments · Studied environments | algorithm performance | Helped |
| Add momentum to TD | TD on Baird's counterexample · Baird's counterexample | convergence | No clear effect |
| Sweep stepsizes, target network refresh rates, and replay buffer sizes for every experiment | DQN hyperparameter selection · Mountain Car | experiments over-reporting average performance for the best tested hyperparamete: 96% | Hurt |
| Tune the default hyperparameters | DQN · Lunar Lander | performance | Helped |
| Use common confidence interval approaches to estimate the mean | DQN · PuddleWorld | mean performance coverage | Hurt |
Surprising findings
- Roughly 96% of 1000 hyperparameter-swept DQN experiments over-report the best configuration's average performance on Mountain Car.
- Momentum does not resolve TD divergence: TD with momentum still diverges on Baird's counterexample.
Limitations
- Cross-validation-like procedure for hyperparameter selection across environments is untested and may have unknown biases, as the authors acknowledge.
- Hyperparameter sweeps are rarely viable for real-world problems unless a simulator is available.
Experimental setup
Datasets: Mountain Car PuddleWorld Lunar Lander Baird's counterexample Maze gridworld
Benchmarks: Mountain Car PuddleWorld Lunar Lander Baird's counterexample Maze gridworld
Metrics: average return tolerance intervals bootstrap confidence intervals
Baselines: DQN Soft Actor-Critic DDPG
Builds on: Expected SARSA DQN Soft Actor-Critic DDPG
Closest papers by meaning
- Position: Benchmarking is Limited in Reinforcement Learning Research (ICML 2024 · similarity 0.84)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.84)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.84)
- BenchMARL: Benchmarking Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.83)
- An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.83)
- Exploiting the Replay Memory Before Exploring the Environment: Enhancing Reinforcement Learning Through Empirical MDP Iteration (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.