Research map / Reinforcement learning / Visual and representation RL

First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Task-inference meta-RL · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes First-Explore, a meta-RL framework that trains separate explore and exploit policies (shared transformer, interleaved cross-entropy updates) and combines them at inference, so cumulative-reward meta-RL no longer gets stuck avoiding exploration on domains requiring sacrificed early reward.

Problem

Cumulative-reward meta-RL methods fail to learn exploration when optimal behavior requires forgoing immediate reward in early episodes, converging to a policy of not exploring even on simple bandit domains.

Approach

Learn two policies: an explore policy trained only to produce episodes that make subsequent exploit episodes higher-return, and an exploit policy trained to maximize episode return given context; after training, choose k explore episodes followed by exploitation at inference.

Key results

• On the deceptive bandit domain First-Explore achieves 127.7 ± 2.0 cumulative reward versus 56.1 ± 12.2 for RL^2, 116.8 for UCB-1 and 123.3 for Thompson Sampling.
• On the deceptive 10-episode Dark Treasure Room (rho=-4) First-Explore achieves 2.0 ± 0.3 versus 0.2 for RL^2 and VariBAD and -0.2 for HyperX.
• On Ray Maze First-Explore achieves 0.47 ± 0.01 mean reward versus 0 for RL^2 and VariBAD and 0.07 for HyperX, near the 0.64 optimum.

Measured effects reported in the paper

Surprising findings

• First-Explore also outperforms the cumulative-reward controls on the non-deceptive domain variants, although the paper only claims benefits in deceptive settings.
• RL^2 and VariBAD converge to never moving on deceptive domains, and HyperX's apparent improvement is an artifact of its exploration bonus being annealed to zero.

Limitations

• Explore policy does not actively explore to enable future exploration; myopic exploration is not necessarily optimal.
• Explore policy may take unsafe actions since it does not avoid negative rewards, raising safety concerns in physical environments.
• Uses a fixed number of exploration episodes k chosen after training, and long-context sequence modelling may be costly.

Experimental setup

Datasets: Bandits with One Fixed Arm (constructed) Dark Treasure Rooms (constructed) Ray Maze (constructed)

Benchmarks: Bandits with One Fixed Arm Dark Treasure Rooms Ray Maze

Metrics: mean cumulative reward standard deviation median reward Mann-Whitney U p-value

Baselines: RL^2 VariBAD HyperX UCB-1 Thompson Sampling Random Action

Builds on: RL^2 VariBAD HyperX GPT-2

Closest papers by meaning

• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices (ICML 2021 · similarity 0.93)
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration (ICML 2021 · similarity 0.91)
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Discovering Temporally-Aware Reinforcement Learning Algorithms (ICLR 2024 · similarity 0.89)
• A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.89)
• MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning (ICLR 2024 · similarity 0.88)