Research map / Reinforcement learning / Policy gradient and actor-critic
Virtual Action Actor-Critic Framework for Exploration (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes virtual action actor-critic (VAAC), which adds a virtual actor that maximizes anticipated novelty of predicted next states to drive exploration via a virtual-policy-entropy-regularized Q function, evaluated on a continuous 4-room maze and SparseMujoco tasks.
Problem
Efficient exploration in reinforcement learning, especially with sparse rewards and high-dimensional state-action spaces, without requiring prior experience in unfamiliar states.
Approach
A virtual actor takes virtual actions to anticipate next-state novelty using a learned dynamics model and novelty module, without environment interaction; its virtual policy entropy is added as a penalty to the actor-critic objective so that high entropy signals no exploration potential.
Key results
- In the continuous 4-room maze without rewards, VAAC visits many more distinct states than SAC and RND over 500k steps (averaged over 30 seeds).
- On SparseMujoco after 1M steps, VAAC reaches max average return of 951.6 on S.HalfCheetah versus 916.3 for SAC and 266.4 for RND.
- VAAC achieves the highest max average return on S.Hopper (894.6) and S.Walker2d (889.5), while on S.Ant RND (670.8) is roughly tied with VAAC (665.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VAAC instead of SAC | SparseMujoco exploration · S.HalfCheetah | max average return: 916.3±37.8 -> 951.6 ± 2.5 | Helped |
| Use VAAC instead of RND | SparseMujoco exploration · S.HalfCheetah | max average return: 266.4±326.2 -> 951.6 ± 2.5 | Helped |
| Use VAAC instead of SAC | SparseMujoco exploration · S.Hopper | max average return: 866.7±14.5 -> 894.6 ± 5.2 | Helped |
| Use VAAC instead of RND | SparseMujoco exploration · S.Hopper | max average return: 113.2±43.0 -> 894.6 ± 5.2 | Helped |
| Use VAAC instead of SAC | SparseMujoco exploration · S.Walker2d | max average return: 873.6±42.1 -> 889.5 ± 11.9 | Helped |
| Use VAAC instead of RND | SparseMujoco exploration · S.Walker2d | max average return: 6.8±6.0 -> 889.5 ± 11.9 | Helped |
| Use VAAC instead of SAC | SparseMujoco exploration · S.Ant | max average return: 584.3±103.5 -> 665.7 ± 78.3 | Helped |
| Use VAAC instead of RND | SparseMujoco exploration · S.Ant | max average return: 670.8 ± 149.9 -> 665.7 ± 78.3 | Hurt |
Surprising findings
- On S.Ant, RND achieves a comparable maximum average return to VAAC, so VAAC does not win on every task.
Limitations
- Performance depends on the reliability of the ANRM and virtual actor, which is not yet studied in depth.
- Training is computationally intensive due to framework complexity.
- Convergence is not theoretically guaranteed.
Experimental setup
Datasets: Continuous 4-room maze SparseMujoco
Benchmarks: Continuous 4-room maze SparseMujoco (S.HalfCheetah, S.Hopper, S.Walker2d, S.Ant)
Metrics: number of state visits maximum average return
Baselines: SAC RND
Builds on: soft actor-critic random network distillation
Closest papers by meaning
- Better Exploration with Optimistic Actor Critic (NeurIPS 2019 · similarity 0.87)
- Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL (NeurIPS 2025 · similarity 0.87)
- Decoupled Policy Actor-Critic: Bridging Pessimism and Risk Awareness in Reinforcement Learning (AAAI 2025 · similarity 0.86)
- Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.85)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.