Research map / Reinforcement learning / Policy gradient and actor-critic
OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OVD-Explorer, a noise-aware optimistic exploration method for continuous control that uses an optimistic value distribution and gradient ascent on an exploration-ability measure, built on distributional SAC.
Problem
Optimism-driven exploration over-explores high-noise (aleatoric) regions in noisy environments, hurting exploration efficiency, and prior noise-aware optimistic methods are limited to discrete control.
Approach
Defines a policy exploration-ability measure as mutual information between the policy and an optimistic upper-bound return distribution (epistemic uncertainty added to the mean, aleatoric variance kept), and derives the behavior policy by gradient ascent on this measure, so exploration is optimistic but discounts high-variance actions.
Key results
- On standard MuJoCo tasks, OVD-Explorer_Q averages 7153.92 vs 6229.00 for DSAC and 5672.92 for SAC over five tasks (10 seeds).
- On five noisy MuJoCo tasks, OVD-Explorer_G and _Q average 422.67 and 422.83 vs 408.25 for DSAC and 270.31 for DOAC.
- On GridChaos standard setting (row s), OVDE averages 59.30 vs 0.00 for DSAC and 3.02 for DOAC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add noise-aware optimistic exploration | DSAC on GridChaos S · GridChaos S | average return: 3.02 -> 59.30 | Helped |
| add noise-aware optimistic exploration | DSAC on GridChaos A-E · GridChaos A-E | average return: 38.42 -> 58.99; 39.78 -> 79.52; 20.59 -> 39.64; 39.99 -> 40.46; 14.60 -> 20.20 | Helped |
| add noise-aware optimistic exploration | DSAC on Ant-v2 · Ant-v2 | average return: 6385.9±1287.2 -> 7382.3±466.6 | Helped |
| add noise-aware optimistic exploration | DSAC on HalfCheetah-v2 · HalfCheetah-v2 | average return: 13348.4±1957.1 -> 16484.3±1373.75 | Helped |
| add noise-aware optimistic exploration | DSAC on Hopper-v2 · Hopper-v2 | average return: 2506.0±390.56 -> 2559.3±384.5 | Hurt |
| use quantile distribution instead of Gaussian distribution for the policy return distribution (ablation) | OVD-Explorer_G on HalfCheetah-v2 · HalfCheetah-v2 | average return: 14796.2±1473.2 -> 16484.3±1373.75 | Helped |
| use quantile distribution instead of Gaussian distribution for the policy return distribution (ablation) | OVD-Explorer_G on N-Pusher-v2 · N-Pusher-v2 | average return: -25.69±3.57 -> -26.13±3.63 | Hurt |
| turn off pessimistic estimation of the value distribution (ablation) | OVD-Explorer on GridChaos F · GridChaos F | average return: 19.84 -> 39.96 | Helped |
| turn off pessimistic estimation of the value distribution (ablation) | OVD-Explorer on GridChaos H · GridChaos H | average return: 40.00 -> 60.00 | Helped |
| set exploration ratio α too small (ablation) | OVD-Explorer_G on Noisy Ant-v2 · Noisy Ant-v2 | average return | Hurt |
| set exploration ratio α too large (ablation) | OVD-Explorer_G on Noisy Ant-v2 · Noisy Ant-v2 | average return | Hurt |
| use OVD-Explorer_Q instead of OVD-Explorer_G (ablation) | OVD-Explorer on standard and noisy tasks · MuJoCo tasks | average return: 7153.92 vs 6723.66; 422.83 vs 422.67 | Mixed |
Surprising findings
- DOAC, which adds optimism without noise awareness, often underperforms DSAC on noisy MuJoCo tasks.
- Turning off the pessimistic estimate (OVDE(m)) performed better in some high-noise-around-goal GridChaos settings, suggesting excessive pessimism is not always needed.
Limitations
- Gains over DSAC on noisy MuJoCo tasks are modest (422.8 vs 408.3 average), and OVDE_G vs OVDE_Q differ by variant.
- Hyperparameters alpha and beta were tuned by grid search, with sensitivity studies on Noisy Ant only.
- Evaluated only on SAC-based implementation and on MuJoCo/GridChaos; extension to discrete or multi-agent tasks is left as future work.
Experimental setup
Datasets: MuJoCo (Ant-v2, HalfCheetah-v2, Hopper-v2, Reacher-v2, InvertedDoublePendulum-v2) noisy MuJoCo variants (N-Ant, N-HalfCheetah, N-Hopper, N-Pusher, N-InvDbPendulum) GridChaos (custom Gym task)
Benchmarks: MuJoCo standard tasks noisy MuJoCo tasks GridChaos
Metrics: average episodic return FRG epoch (first epoch reaching goal)
Baselines: SAC DSAC DOAC DSAC+RND IDS
Builds on: SAC DSAC OAC QR-DQN clipped double Q-learning
Closest papers by meaning
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.90)
- Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling (ICLR 2025 · similarity 0.88)
- Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL (NeurIPS 2025 · similarity 0.87)
- Minimax Optimal Reinforcement Learning with Quasi-Optimism (ICLR 2025 · similarity 0.86)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.86)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.