Research map / Reinforcement learning / Policy gradient and actor-critic
Adaptive Exploration for Data-Efficient General Value Function Evaluations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GVFExplorer, which adaptively learns a single behavior policy that minimizes the summed return variance across GVFs (via TD-based variance estimation) to data-efficiently evaluate many GVFs in parallel.
Problem
Evaluating multiple General Value Functions in parallel with off-policy learning is data-inefficient when the behavior policy is fixed, round-robin, or pre-collected, because large value errors arise where return variance is high and coverage is poor.
Approach
Derives a closed-form behavior policy update that places action probabilities proportional to the square root of the summed squared-policy-weighted return variances across GVFs, with variance estimated by an Expected-Sarsa TD-style recursion; proves each update does not increase total variance.
Key results
- In the 20x20 gridworld with two distinct policies and distinct cumulants, GVFExplorer reaches avg MSE 0.04 at 2e6 steps versus 0.44 for MixPol and 0.47 for round-robin (about 91% improvement over the best baseline).
- For 40 GVFs, average MSE is 1.66 versus 2.35 for round-robin (29% improvement).
- The paper proves that each behavior policy update does not increase the summed return variance across GVFs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GVFExplorer instead of round-robin sampling | Parallel GVF evaluation · Gridworld, distinct policies, same cumulant | average MSE: 1.15 -> 0.24; 79% | Helped |
| Use GVFExplorer instead of MixPol sampling | Parallel GVF evaluation · Gridworld, distinct policies, distinct cumulants | average MSE: 0.44 -> 0.04; 91% | Helped |
| Use GVFExplorer instead of UniformPol sampling | Parallel GVF evaluation · FourRooms, non-stationary cumulants | average MSE: 0.9 -> 0.46; 48% | Helped |
| Use GVFExplorer instead of round-robin sampling | 40-GVF evaluation · Gridworld, 40 distinct GVFs | average MSE: 2.35 -> 1.66; 29% | Helped |
| Use coarser state-feature approximations (ablation) | GVFExplorer · Gridworld, distinct policies and distinct cumulants | MSE | Hurt |
| Use a very coarse feature approximation (grouping factor 4) instead of a finer approximation (ablation) | GVFExplorer · Gridworld, distinct policies and distinct cumulants | MSE relative to UniformPol | Hurt |
| Use semi-greedy target policies instead of the less directed target policies (ablation) | GVFExplorer · Gridworld, distinct policies and distinct cumulants | average MSE | No clear effect |
| Use Expected Sarsa updates instead of importance-sampling-corrected TD updates (ablation) | GVFExplorer · FourRooms, distractor and drifter cumulants | average MSE | Helped |
| Add Prioritized Experience Replay (PER) (ablation) | GVFExplorer · Continuous grid environment with nonlinear function approximation | MSE | Helped |
| Prioritize replay using variance-function TD error instead of Q-function TD error (ablation) | GVFExplorer with PER · Continuous grid environment with nonlinear function approximation | MSE | Hurt |
| Increase drifter cumulant variability (ablation) | GVFExplorer · FourRooms, non-stationary cumulant | average MSE | Hurt |
Surprising findings
- Placing prioritized experience replay on the variance-function TD error performed worse than prioritizing on the Q-function TD error.
- In near-greedy target policy settings, round-robin matched or beat GVFExplorer, which needed more samples to converge.
Limitations
- Increased time complexity from learning separate value and variance networks.
- Not evaluated where cumulant value ranges differ greatly, which can cause oversampling of high-valued regions.
- Only policy evaluation is studied; control with unknown target policies is left for future work.
Experimental setup
Datasets: DM-Control Walker DM-Control Cheetah FourRooms
Benchmarks: 20x20 gridworld GVF MSE FourRooms non-stationary cumulant MSE continuous 1x1 grid with function approximation DM-Control Walker and Cheetah GVF MSE
Metrics: average MSE across GVFs
Baselines: Round-robin policy sampling Mixture policy sampling Uniform policy Successor Representation behavior policy (SR) Behavior policy search (BPS)
Builds on: General Value Functions Expected Sarsa Soft Actor-Critic Successor Representation exploration (Janz et al.) Behavior policy search (Hanna et al.) Prioritized Experience Replay
Closest papers by meaning
- A Unifying Framework of Off-Policy General Value Function Evaluation (NeurIPS 2022 · similarity 0.90)
- Gamma-Nets: Generalizing Value Estimation over Timescale (AAAI 2020 · similarity 0.87)
- Uncertainty-Aware Reward-Free Exploration with General Function Approximation (ICML 2024 · similarity 0.87)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.87)
- Reward-Free Kernel-Based Reinforcement Learning (ICML 2024 · similarity 0.85)
- Kernel-Based Function Approximation for Average Reward Reinforcement Learning: An Optimist No-Regret Algorithm (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.