Research map / Reinforcement learning / Policy gradient and actor-critic
Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes policy reproducibility (return dispersion across rollouts), quantified with MAD, and recommends the Lower Confidence Bound of return as an evaluation metric with a preference parameter alpha trading performance against reproducibility, showing on Brax Ant and HalfCheetah that ES yields more reproducible policies than SAC and TD3.
Problem
Standard RL evaluation summarises stochastic policy return distributions only by expected return, which ignores the spread of returns across rollouts (policy reproducibility) and leaves no way to trade off performance against reproducibility.
Approach
Define policy reproducibility as the statistical dispersion of a policy's return distribution, measure it robustly with median absolute deviation, and evaluate policies with LCB = performance − alpha × MAD, where alpha is a user-chosen evaluation-only preference parameter that recovers expected return at alpha=0.
Key results
- On Ant with init-state noise, SAC has LCB 7269.30 at alpha=0 but drops to 3646.34 at alpha=2.0e3, while R-ES has 4865.26 at that alpha, showing reproducibility-aware ranking changes.
- ES and R-ES yield lower return MAD than SAC and TD3 across noise types in Ant (e.g., parameter noise MAD 0.02 and 0.00 vs 1.74 and 1.20 for SAC and TD3).
- ES has lower expected return than SAC at alpha=0 (e.g., Ant init-state 5306 vs 7269), reflecting a performance cost of higher reproducibility.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize a weighted sum of performance and reproducibility with R-ES instead of standard ES (ablation) | ES · Ant and HalfCheetah, all five noise configurations | MAD: Ant: Action 0.87 -> 0.55; Init-State 0.31 -> 0.11; Obs 0.72 -> 0.15; Param 0.02 -> 0.00; Reward 0.34 -> 0.10. HalfCheeta | Helped |
| Use ES-based approaches rather than SAC | SAC · Ant and HalfCheetah, uncertain environments | MAD | Helped |
| Increase the LCB trade-off parameter from α= 0.0 to α=2.0×10^3 (Ant) or α=2.0×10^2 (HalfCheetah) | SAC and TD3 policies · Ant and HalfCheetah, all five noise configurations | LCB | Hurt |
| Increase the LCB trade-off parameter from α= 0.0 to α=2.0×10^3 (Ant) or α=2.0×10^2 (HalfCheetah) | ES and R-ES policies · Ant and HalfCheetah, uncertain environments | LCB | No clear effect |
| Use median rather than expected return to estimate performance in LCB (ablation) | LCB evaluation · Ant Init-State noise | LCB | No clear effect |
| Use IQR rather than MAD to quantify reproducibility in LCB (ablation) | LCB evaluation · Ant and HalfCheetah, uncertain environments | LCB | No clear effect |
| Use standard deviation rather than MAD to quantify reproducibility in LCB (ablation) | LCB evaluation · Ant and HalfCheetah, uncertain environments | LCB | No clear effect |
| Use IQR rather than MAD to measure return dispersion (ablation) | Policy reproducibility evaluation · Ant and HalfCheetah | confidence interval width | Hurt |
| Use the state-marginal representation rather than the behaviour descriptor (ablation) | Behaviour reproducibility evaluation · Ant and HalfCheetah | behavioural MAD | Hurt |
| Use the behaviour descriptor representation rather than the state-marginal representation (ablation) | Behaviour reproducibility evaluation · Ant and HalfCheetah | behavioural reproducibility | Helped |
Surprising findings
- Behavioural reproducibility of ES is less strong on HalfCheetah than on Ant, although R-ES performs consistently well.
Limitations
- Evaluation limited to two continuous-control tasks (Ant and HalfCheetah) in Brax
- Reproducibility measured only in-distribution, out-of-distribution cases such as sim-to-real not studied
- Behavioural reproducibility results are sensitive to the choice of behaviour representation and distance
Experimental setup
Datasets: Brax Ant Brax HalfCheetah
Benchmarks: Brax Ant Brax HalfCheetah
Metrics: MAD IQR standard deviation LCB behavioural MAD
Baselines: SAC TD3 ES
Builds on: Bayesian optimisation lower confidence bound OpenAI-ES TD3 SAC rliable
Closest papers by meaning
- Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation (NeurIPS 2025 · similarity 0.90)
- Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning (AAAI 2026 · similarity 0.90)
- Evaluating the Performance of Reinforcement Learning Algorithms (ICML 2020 · similarity 0.90)
- Doubly Optimal Policy Evaluation for Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.88)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.