Research map / Reinforcement learning / Visual and representation RL
KEA: Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KEA, which pairs a novelty-augmented SAC agent (RND or NovelD) with a standard SAC agent and switches between them based on state novelty, improving sparse-reward exploration on a 2D navigation task, DeepSea, and DeepMind Control sparse tasks.
Problem
Novelty-based exploration combined with SAC's stochastic policy and entropy regularization can interfere, causing the agent to repeatedly revisit high-novelty states and collect redundant experience under sparse rewards.
Approach
Integrates a standard SAC agent (trained only on the sparse extrinsic reward, with frozen learning until extrinsic reward is seen) alongside a novelty-augmented SAC agent, sharing a replay buffer and using a novelty-threshold switching rule to choose which policy acts.
Key results
- On 2D navigation, KEA-RND-SAC reached 0.403 ± 0.042 episodic return versus 0.235 ± 0.184 for RND-SAC after 300K steps.
- On DeepSea size 30, KEA-RND-SAC achieved 0.54 ± 0.32 versus 0.35 ± 0.44 for RND-SAC and 0.42 ± 0.33 for SOFE-DQN.
- On DM Control sparse tasks after 500K steps, KEA-RND-SAC improved over RND-SAC by 119% on Walker Run Sparse (629.74 vs 287.65) and 51% on Cheetah Run Sparse (773.76 vs 512.02).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KEA coordination to RND-SAC | SAC · 2D Navigation | mean episodic return: 0.235 ± 0.184 -> 0.403 ± 0.042 | Helped |
| Add KEA coordination to NovelD-SAC | SAC · 2D Navigation | learning speed: 190,000 -> 250,000 steps to achieve a similar return | Helped |
| Add KEA coordination to RND-SAC | SAC · DeepSea 14 | average return: 0.92 ± 0.13 -> 0.99 ± 0.01 | Helped |
| Add KEA coordination to RND-SAC | SAC · DeepSea 20 | average return: 0.89 ± 0.09 -> 0.92 ± 0.05 | Helped |
| Add KEA coordination to RND-SAC | SAC · DeepSea 24 | average return: 0.67 ± 0.35 -> 0.81 ± 0.18 | Helped |
| Add KEA coordination to RND-SAC | SAC · DeepSea 30 | average return: 0.35 ± 0.44 -> 0.54 ± 0.32 | Helped |
| Add KEA coordination to RND-SAC | SAC · Walker Run Sparse | mean episodic return: 287.65 ± 334.12 -> 629.74 ± 196.75 | Helped |
| Add KEA coordination to RND-SAC | SAC · Cheetah Run Sparse | mean episodic return: 512.02 ± 466.26 -> 773.76 ± 162.74 | Helped |
| Add KEA coordination to RND-SAC | SAC · Reacher Hard Sparse | mean episodic return: 790.32 ± 143.26 -> 874.61 ± 94.58 | Helped |
| Add KEA coordination to NovelD-SAC | SAC · Walker Run Sparse | mean episodic return: 553.26 ± 191.03 -> 706.47 ± 389.23 | Helped |
| Add KEA coordination to NovelD-SAC | SAC · Cheetah Run Sparse | mean episodic return: 647.29 ± 382.58 -> 734.67 ± 316.95 | Helped |
| Add KEA coordination to NovelD-SAC | SAC · Reacher Hard Sparse | mean episodic return: 860.40 ± 76.15 -> 837.12 ± 68.95 | Hurt |
Surprising findings
- KEA-RND-SAC did not improve over RND-DQN because epsilon-greedy exploration is independent of novelty, so the two strategies do not interact.
- On the 2D navigation task, KEA-NovelD-SAC matched NovelD-SAC in final return (0.604 vs 0.607), with gains only in convergence speed.
Limitations
- Restricted to off-policy learning because the standard agent shares experience with the target policy.
- Switching threshold sigma is a sensitive hyperparameter that varies by task (1.0 for 2D navigation, 0.75 for DM Control).
- Evaluation is limited to small-scale tasks (2D grid, DeepSea, three DM Control tasks) with five seeds.
Experimental setup
Datasets: DeepSea DeepMind Control Suite (Walker Run Sparse, Cheetah Run Sparse, Reacher Hard Sparse) custom 2D navigation (41x41 grid)
Benchmarks: 2D Navigation DeepSea DeepMind Control Suite sparse tasks
Metrics: mean episodic return standard deviation over five seeds
Baselines: SAC RND-SAC NovelD-SAC DeRL SOFE DQN Soft Q-Learning
Builds on: Soft Actor-Critic Random Network Distillation NovelD
Closest papers by meaning
- Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration (NeurIPS 2025 · similarity 0.85)
- Improving Intrinsic Exploration by Creating Stationary Objectives (ICLR 2024 · similarity 0.85)
- CIG: Exploration via Conditional Information Gain (NeurIPS 2026 · similarity 0.85)
- Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL (NeurIPS 2025 · similarity 0.85)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.84)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.