Research map / Reinforcement learning / Visual and representation RL
Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes regret in realtime RL with asynchronous default behavior and proposes staggered asynchronous inference processes so large (up to 1B-parameter) DQN/Rainbow ResNet agents can act at consistent intervals in Game Boy (Pokémon, Tetris) and Atari realtime simulations.
Problem
Large neural network policies have long inference times, so in realtime environments that keep evolving during inference, sequential interaction forces inaction (default no-op actions) and persistent regret.
Approach
Decompose realtime regret into learning, inaction, and delay terms, and use multiple staggered asynchronous inference processes (with max-time or expected-time waiting schemes) so that actions are taken at consistent intervals, reducing the interaction time roughly linearly with the number of processes.
Key results
- Staggered asynchronous inference maintains Tetris performance for models above 1B parameters where sequential interaction cannot surpass random performance for |θ| ≥ 1M.
- The number of inference processes needed to eliminate inaction, N*_I, scales roughly linearly with both inference time and parameter count.
- Asynchronous interaction reaches superior Pokémon Blue battle and catching progress at 100M parameters compared with sequential baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use staggered asynchronous inference and learning instead of sequential interaction | DQN ResNet agents · Pokémon Blue battles and catching | progress over time | Helped |
| Use staggered asynchronous inference instead of a single inference process | DQN ResNet agents · Tetris | average episodic return | Helped |
| Use staggered asynchronous inference instead of a single inference process | DQN ResNet agents · Boxing, Krull, and Name This Game | average episodic return | Helped |
| Use staggered asynchronous inference instead of a single inference process | Rainbow agents · Boxing, Krull, and Name This Game | average episodic return | Helped |
| Increase model size while using sequential interaction | DQN ResNet agents · Tetris | average episodic return | Hurt |
| Increase model size while using staggered asynchronous inference | DQN ResNet agents · Tetris | average episodic return | Helped |
| Increase inference time and use staggered asynchronous inference | DQN ResNet agents · Krull | average episodic return | Helped |
| Increase inference time and use staggered asynchronous inference | DQN ResNet agents · Boxing | average episodic return | Hurt |
| Use a 30-convolutional-layer ResNet instead of a 15-convolutional-layer ResNet (ablation) | DQN Atari agents · Atari | average episodic return | Mixed |
| Use round-robin asynchronous learning with a larger batch size (ablation) | DQN ResNet agents · Game Boy environments | learning throughput | Helped |
| Use Algorithm [ref] (maximum-time inference staggering) instead of Algorithm [ref] (expected-time staggering) (ablation) | Asynchronous inference processes · Game Boy inference scaling with ε=0.5 | inference process requirement N*I | Helped |
| Compare progress per non-noop action between asynchronous and sequential interaction | DQN agents · Pokémon Blue battles and catching | progress per non-noop action | No clear effect |
Surprising findings
- Sequential interaction cannot surpass random performance in Tetris for models with at least 1M parameters.
- Pokémon game progress per non-noop action is not statistically different between sequential and asynchronous methods, suggesting the game does not reward acting every frame.
Limitations
- Performance remains well below human level in Pokémon and Tetris because models are trained from scratch for limited time.
- Assumes ideal dedicated-CPU hardware; memory bandwidth and GPU co-location tradeoffs are not analyzed.
- Results on human-level policies and stochastic environments such as Boxing may not generalize.
Experimental setup
Datasets: Pokémon Blue (Game Boy via Gymnasium Retro) Tetris (Game Boy via Gymnasium Retro) Boxing (Atari) Krull (Atari) Name This Game (Atari)
Benchmarks: Pokémon Blue battle encounters Pokémon Blue catching encounters Tetris Boxing Krull Name This Game
Metrics: battles won over time wild Pokémon caught over time average episodic return N*_I scaling N*_L scaling
Baselines: sequential interaction DQN parallel learning baseline single inference process human performance
Builds on: DQN Rainbow Gymnasium Retro round-robin asynchronous SGD random delay MDP
Closest papers by meaning
- Pausing Policy Learning in Non-stationary Reinforcement Learning (ICML 2024 · similarity 0.90)
- Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies (NeurIPS 2025 · similarity 0.89)
- Sleeping Reinforcement Learning (ICML 2025 · similarity 0.88)
- Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds (ICLR 2025 · similarity 0.87)
- Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.87)
- Handling Delay in Real-Time Reinforcement Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.