Research map / Reinforcement learning / Policy gradient and actor-critic
CALE: Continuous Arcade Learning Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CALE, a wrapper around the Atari 2600 emulator (Stella) in the ALE that replaces the 18 discrete joystick actions with a continuous 3-D action space, and provides SAC/PPO/SAC-D baselines on discrete-vs-continuous comparisons.
Problem
Standard ALE benchmarks only support discrete-action agents, so continuous-control methods (e.g., PPO, SAC) are rarely evaluated on Atari, leaving discrete and continuous agents without a unified comparison benchmark.
Approach
Map joystick motion to a continuous action space (polar coordinates r, theta plus a fire dimension) and discretize it via a threshold tau into the nine joystick positions and fire events, so the same Stella events and game dynamics are used for both discrete and continuous agents.
Key results
- SAC baseline on CALE reaches only about 0.4 IQM human-normalized score, far below DQN and Data-Efficient Rainbow on the ALE.
- Standard SAC sampling outperforms epsilon-greedy exploration in the 200M regime.
- Discrete-action SAC on the ALE dramatically underperforms continuous-action SAC on CALE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use standard SAC exploration instead of ε-greedy exploration (ablation) | SAC on CALE · 200 million training regime | performance | Helped |
| Use the SAC convolutional encoder instead of the DQN convolutional encoder (ablation) | SAC on CALE · 100K and 200M training regimes | performance | Helped |
| Use SAC on CALE instead of DQN on ALE | SAC · 200 million training regime | aggregate performance | Hurt |
| Use SAC on CALE instead of Data-Efficient Rainbow on ALE | SAC · Atari 100K | aggregate performance | Hurt |
| Use categorical actions with SAC on ALE instead of continuous actions with SAC on CALE (ablation) | SAC · Atari | performance | Hurt |
| Use SAC instead of DQN | SAC and DQN · Asteroids, Bowling, Centipede | performance | Helped |
| Use SAC instead of DQN | SAC and DQN · Asterix, Boxing, MsPacman, Pong | performance | No clear effect |
| Use SAC instead of DQN | SAC and DQN · BankHeist, Breakout, SpaceInvaders | performance | Hurt |
| Increase the joystick threshold τ (ablation) | CALE agents · 100K and 200M training regimes | performance | Hurt |
| Use ε-greedy exploration instead of sampling from the categorical distribution (ablation) | SAC-D on ALE · Atari | performance | Helped |
Surprising findings
- Discrete-action SAC on the ALE underperforms continuous-action SAC on CALE, which the authors find surprising.
- Standard sampling-based exploration outperforms epsilon-greedy for continuous SAC, contrary to the common discrete-action practice.
Limitations
- Only SAC and PPO from Dopamine were evaluated as continuous agents, limiting baseline breadth.
- No minimal action set in CALE, disadvantaging continuous agents on games where ALE uses a reduced action set.
- SAC was not tuned for CALE, so the baseline gap may reflect tuning rather than inherent limits.
Experimental setup
Datasets: Arcade Learning Environment games (60 games; Atari 100K 26-game subset) RL Unplugged ALE dataset ExoRL DMC dataset
Benchmarks: Arcade Learning Environment (60 games, 200M frames) Atari 100K (26 games)
Metrics: interquartile mean (IQM) of human-normalized score per-game return
Baselines: DQN Data-Efficient Rainbow SAC-D PPO DER
Builds on: Arcade Learning Environment Stella emulator Soft Actor-Critic Dopamine Data-Efficient Rainbow
Closest papers by meaning
- Atari-5: Distilling the Arcade Learning Environment down to Five Games (ICML 2023 · similarity 0.85)
- AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents (NeurIPS 2026 · similarity 0.80)
- Learning to Play Atari in a World of Tokens (ICML 2024 · similarity 0.80)
- BlendRL: A Framework for Merging Symbolic and Neural Policy Learning (ICLR 2025 · similarity 0.80)
- Using Unity to Help Solve Reinforcement Learning (NeurIPS 2024 · similarity 0.80)
- Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning (ICML 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.