Research map / Reinforcement learning / Visual and representation RL
EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends EfficientZero to continuous and discrete control, visual and proprioceptive inputs, using sampling-based Gumbel tree search and search-based value estimation with a mixed value target, outperforming DreamerV3 and other SOTA in sample-limited benchmarks.
Problem
Sample-efficient RL that performs consistently well across discrete and continuous action spaces and both visual and low-dimensional inputs; prior methods (MCTS-based EfficientZero, TD-MPC, DreamerV3) each lack consistent performance across domains.
Approach
Replace MCTS with a sampling-based Gumbel search that mixes actions from the current Gaussian policy and a flattened prior (guaranteeing policy improvement in continuous spaces), and use a mixed value target that switches between multi-step TD for fresh data/early training and search-based value estimation for stale transitions.
Key results
- Atari 100k: normalized mean 2.428 and median 1.286, above BBF (2.247 / 0.917) and EfficientZero (1.945 / 1.090).
- Vision Control: mean score 726.1 vs DreamerV3 498.5 (about 45% higher), best on 16 of 20 tasks.
- Proprio Control 50k mean 723.2 vs TD-MPC2 740.9 (comparable), with far fewer imagined latent states (32 vs 9216 for MPPI).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EZ-V2 instead of BBF | Atari 100k · Atari 100k | Normed Mean: 2.247 -> 2.428 | Helped |
| use EZ-V2 instead of EfficientZero | Atari 100k · Atari 100k | Normed Mean: 1.945 -> 2.428 | Helped |
| use EZ-V2 instead of TD-MPC2 | Proprio Control · Proprio Control 100k | Mean: 740.9 -> 723.2 | Hurt |
| use EZ-V2 instead of DreamerV3 | Vision Control · Vision Control 200k | Mean: 498.5 -> 726.1 | Helped |
| add the mixed value target instead of using multi-step TD (ablation) | EZ-V2 · Proprio Control 50-100k: Acrobot Swingup | score: 256.7 -> 297.7 | Helped |
| add the mixed value target instead of using GAE (ablation) | EZ-V2 · Proprio Control 50-100k: Acrobot Swingup | score: 275.1 -> 297.7 | Helped |
| use sampling-based Gumbel search instead of Sample MCTS (ablation) | EZ-V2 · Proprio Control 50-100k: Acrobot Swingup | score: 248.1 -> 297.7 | Helped |
| add the mixed value target instead of using multi-step TD (ablation) | EZ-V2 · Vision Control 100-200k: Acrobot Swingup | score: 186.3 -> 231.8 | Helped |
| add the mixed value target instead of using GAE (ablation) | EZ-V2 · Vision Control 100-200k: Acrobot Swingup | score: 122.0 -> 231.8 | Helped |
| use sampling-based Gumbel search instead of Sample MCTS (ablation) | EZ-V2 · Vision Control 100-200k: Acrobot Swingup | score: 73.9 -> 231.8 | Helped |
| add the mixed value target instead of using multi-step TD (ablation) | EZ-V2 · Atari 100k: Asterix | score: 26023.1 -> 61810.0 | Helped |
| add the mixed value target instead of using GAE (ablation) | EZ-V2 · Atari 100k: Asterix | score: 22816.7 -> 61810.0 | Helped |
Surprising findings
- Sampled-MCTS-style search underperforms the proposed search even with fewer simulations (32 vs 50), and 8 simulations already surpass Sample MCTS on Quadruped Walk.
- On Atari 100k, EZ-V2 scores 0.0 on Freeway and lower than BBF on several games such as ChopperCommand (1196.6 vs 7549.3).
Limitations
- Gains over the best baselines are small on some proprio tasks (e.g., TD-MPC2 mean 740.9 vs EZ-V2 723.2), and the performance gap narrows with more data.
- Evaluated only in simulation (Atari and DMControl); real-world robotics is left as future work.
- Some Atari games (e.g., Freeway, ChopperCommand) remain weak relative to BBF.
Experimental setup
Datasets: Atari 100k DeepMind Control Suite
Benchmarks: Atari 100k DMControl Proprio Control 50k DMControl Proprio Control 100k DMControl Vision Control 100k DMControl Vision Control 200k
Metrics: human-normalized mean score human-normalized median score episode return
Baselines: DreamerV3 TD-MPC2 EfficientZero BBF SAC DrQ-v2 CURL SPR Sample MCTS
Builds on: EfficientZero Gumbel MuZero Sampled MuZero MuZero SimSiam
Closest papers by meaning
- Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.88)
- FASTER: Value-Guided Sampling for Fast RL (NeurIPS 2026 · similarity 0.88)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.88)
- Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity (ICLR 2024 · similarity 0.88)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.