Research map / Reinforcement learning / MDPs and planning
Uncertainty-Aware Reward-Free Exploration with General Function Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GFA-RFE, a model-free reward-free exploration algorithm with uncertainty-aware intrinsic reward and variance-weighted regression for general function approximation, with theoretical sample complexity bounds and DeepMind Control Suite (URLB) experiments.
Problem
Reward-free exploration in RL with general function approximation: heterogeneous uncertainty across collected samples reduces sample efficiency, and existing bounds are loose or computationally costly.
Approach
Uses a generalized-eluder-dimension-based uncertainty estimate (approximated by ensemble Q-variance) as the intrinsic reward and as a bonus, and weights the Bellman regression by estimated aleatoric plus epistemic variance per sample, yielding a model-free algorithm with improved sample complexity.
Key results
- Theoretical sample complexity of Õ(H^2 log N_F(ε) dim(F)/ε^2) episodes, claimed to outperform existing reward-free RL bounds under general function approximation.
- On URLB Walker, GFA-RFE attains 925±50 on stand (best) and 554±64 on flip (second-best behind Disagreement's 570±32).
- On URLB Quadruped, GFA-RFE is competitive but not uniformly best; e.g., 719±68 on jump vs. 829±22 for Disagreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GFA-RFE instead of the strongest baseline, Disagreement | reward-free exploration · Walker Flip | cumulative reward: 570 $\pm$ 32 -> 554 $\pm$ 64 | No clear effect |
| Use GFA-RFE instead of the strongest baseline, Disagreement | reward-free exploration · Walker Run | cumulative reward: 340 $\pm$ 37 -> 339 $\pm$ 34 | No clear effect |
| Use GFA-RFE instead of the strongest baseline, APT | reward-free exploration · Walker Stand | cumulative reward: 891 $\pm$ 62 -> 925 $\pm$ 50 | Helped |
| Use GFA-RFE instead of the strongest baseline, Disagreement | reward-free exploration · Walker Walk | cumulative reward: 851 $\pm$ 63 -> 826 $\pm$ 89 | No clear effect |
| Use GFA-RFE instead of the strongest baseline, Disagreement | reward-free exploration · Quadruped Run | cumulative reward: 524 $\pm$ 24 -> 460 $\pm$ 36 | Hurt |
| Use GFA-RFE instead of the strongest baseline, Disagreement | reward-free exploration · Quadruped Jump | cumulative reward: 829 $\pm$ 22 -> 719 $\pm$ 68 | Hurt |
| Use GFA-RFE instead of the strongest baseline, Disagreement | reward-free exploration · Quadruped Stand | cumulative reward: 953 $\pm$ 16 -> 867 $\pm$ 61 | Hurt |
| Use GFA-RFE instead of the strongest baseline, RND | reward-free exploration · Quadruped Walk | cumulative reward: 820 $\pm$ 94 -> 726 $\pm$ 146 | Hurt |
Surprising findings
- Disagreement, APT and RND are the strongest baselines; GFA-RFE is compatible or superior rather than uniformly best.
Limitations
- Experiments only on state-based URLB Walker and Quadruped, not pixel-based or broader domains.
- Practical implementation approximates the theoretical uncertainty oracle with ensemble Q-variance, a heuristic departure from the theory.
- Gains over strong baselines are mixed; several tasks show GFA-RFE behind the best method.
Experimental setup
Datasets: DeepMind Control Suite via Unsupervised RL Benchmark (URLB) - Walker, Quadruped
Benchmarks: URLB Walker tasks (flip, run, stand, walk) URLB Quadruped tasks (run, jump, stand, walk)
Metrics: cumulative reward
Baselines: ICM APT DIAYN APS Disagreement SMM RND
Builds on: reward-free exploration (RF-Express / SSTP line) weighted regression (Zhou et al. 2023 style) sensitivity-based exploration (Zhang et al. 2022 style) URLB
Closest papers by meaning
- Reward-Free Exploration for Reinforcement Learning (ICML 2020 · similarity 0.89)
- Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-Free RL (ICLR 2023 · similarity 0.89)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.88)
- Random Latent Exploration for Deep Reinforcement Learning (ICML 2024 · similarity 0.88)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.88)
- Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.