Research map / Reinforcement learning / Visual and representation RL
Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoAct TD learning, which takes the action minimizing the Q-function with probability epsilon during experience collection, and shows it raises temporal-difference magnitude and improves sample efficiency on the Atari 100K benchmark.
Problem
Sample inefficiency of deep RL in high-dimensional MDPs, where common exploration such as epsilon-greedy is simple and experience collection is not well targeted.
Approach
Replace the random exploratory action in epsilon-greedy with the counteractive action that minimizes the state-action value function, arguing that this yields higher temporal-difference transitions at zero extra computational cost, and apply it as a drop-in change to TD-based algorithms.
Key results
- CoAct TD improves human normalized median on ALE 100K to 0.0927 versus 0.0377 for epsilon-greedy with DDQN, reported as a 248% improvement.
- Versus NoisyNetworks (median 0.0457), CoAct TD is reported as a 204% improvement.
- Median normalized TD gain for CoAct TD is up to 25% larger than epsilon-greedy across ALE 100K games.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ε-greedy exploration with CoAct TD counteractive actions | DDQN · ALE 100K | human normalized median: 0.0377±0.0031 -> 0.0927±0.0050; 248% | Helped |
| Replace NoisyNetworks exploration with CoAct TD counteractive actions | DDQN · ALE 100K | human normalized median: 0.0457±0.0035 -> 0.0927±0.0050; 204% | Helped |
| Replace ε-greedy exploration with CoAct TD counteractive actions | DDQN · ALE 100K | human normalized 20th percentile: 0.0056±0.0017 -> 0.0145±0.0003 | Helped |
| Replace NoisyNetworks exploration with CoAct TD counteractive actions | DDQN · ALE 100K | human normalized 20th percentile: 0.0102±0.0018 -> 0.0145±0.0003 | Helped |
| Replace ε-greedy exploration with CoAct TD counteractive actions | DDQN · ALE 100K | human normalized 80th percentile: 0.2942±0.0233 -> 0.3762±0.0137 | Helped |
| Replace NoisyNetworks exploration with CoAct TD counteractive actions | DDQN · ALE 100K | human normalized 80th percentile: 0.1913±0.0144 -> 0.3762±0.0137 | Helped |
| Use counteractive actions instead of ε-greedy actions | CoAct TD · ALE 100K | normalized temporal difference gain: up to 25 percent | Helped |
Surprising findings
- Choosing the action that minimizes the Q-function, rather than a random one, is claimed to increase TD error, which runs counter to standard intuition.
- NoisyNetworks, a more expensive exploration method, performs only modestly above epsilon-greedy on the 100K benchmark.
Limitations
- Theoretical results rely on assumptions (eta-uninformed and delta-smooth Q-functions) that are stated for random initialization and may not hold after training.
- Gains are reported mainly as aggregate medians; the abstract and text do not give per-game variance beyond standard error.
Experimental setup
Datasets: Arcade Learning Environment 100K benchmark Arcade Learning Environment 200M frames
Benchmarks: Arcade Learning Environment 100K Arcade Learning Environment 200M frames chain MDP
Metrics: human normalized median score 20th and 80th percentile human normalized score temporal difference magnitude normalized TD gain
Baselines: epsilon-greedy NoisyNetworks UCB QRDQN DDQN
Builds on: DDQN QRDQN prioritized experience replay epsilon-greedy NoisyNetworks
Closest papers by meaning
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- The Effective Horizon Explains Deep RL Performance in Stochastic Environments (ICLR 2024 · similarity 0.87)
- Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction (AAAI 2024 · similarity 0.87)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Oracle-Efficient Reinforcement Learning for Max Value Ensembles (NeurIPS 2024 · similarity 0.87)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.