Research map / Reinforcement learning / Visual and representation RL

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Combines six post-Rainbow improvements (Impala+adaptive maxpooling, spectral normalization, IQN, Munchausen, vectorized environments) into a Rainbow-style DQN agent that reaches SOTA non-recurrent Atari-60 IQM of 7.4 at 200M frames on a desktop PC.

Problem

State-of-the-art deep RL agents require large distributed compute or days of training, making high-performance RL inaccessible to small labs and hobbyists; the paper targets high performance with low walltime on a single desktop.

Approach

Extends Rainbow DQN by adding six recent improvements (Impala CNN with width scale 2 and adaptive maxpooling, spectral normalization, IQN replacing C51, Munchausen RL replacing Double DQN, and 64 vectorized environments with large batches), tuned for low walltime, and analyzes each component's effect on performance, action gaps, policy churn, dormant neurons and weight norms.

Key results

• BTR achieves an Atari-60 IQM of 7.4 (7.361 in appendix table) at 200M frames, versus Rainbow DQN's 1.9, and exceeds human performance on 52 of 60 games.
• BTR trains 200M Atari frames in about 11.5 hours on an RTX 4090 desktop, versus Rainbow DQN's roughly 35 hours.
• On Procgen, BTR outperforms Rainbow DQN + Impala (4x width) while using about 8 hours of walltime versus 41.

Measured effects reported in the paper

Surprising findings

• Simply adding Impala to Rainbow does not reproduce the gains (6.3 IQM vs 7.7 on Atari-5), so the combination matters.
• Removing Impala collapses performance (e.g., Phoenix score 21k vs 330k with full BTR at epsilon=0).

Limitations

• Does not include an explicit exploration mechanism, so it struggles on hard-exploration games such as Montezuma's Revenge (scored 0).
• Procgen performance does not compete with state-of-the-art, as the method was not targeted at procedurally generated environments.
• Ablation statistics on Atari-5 use few seeds (3-4), and early design decisions were tested on single runs.

Experimental setup

Datasets: Atari-60 Atari-5 Procgen Super Mario Galaxy Mario Kart Wii Mortal Kombat

Benchmarks: Atari-60 Atari-5 Procgen Super Mario Galaxy Mario Kart Wii Mortal Kombat

Metrics: interquartile mean human-normalized score median mean optimality gap walltime action gap policy churn dormant neurons SRank L2 norm

Baselines: Rainbow DQN DQN (Nature) Munchausen-IQN Fast and Efficient Rainbow MEME Dreamer-v3 BBF EfficientZero-v2 PQN Rainbow DQN + Impala

Builds on: Rainbow DQN Munchausen RL IMPALA Implicit Quantile Networks Spectral Normalization

Closest papers by meaning

• Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research (ICML 2021 · similarity 0.91)
• Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.84)
• Deep Double Q-learning (NeurIPS 2026 · similarity 0.84)
• Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.83)
• Hadamax Encoding: Elevating Performance in Model-Free Atari (NeurIPS 2025 · similarity 0.82)
• Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder (AAAI 2025 · similarity 0.82)