Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Intelligent Go-Explore, which replaces Go-Explore's handcrafted heuristics (state selection, action selection, archive filtering) with GPT-4 judgments, and evaluates it on Game of 24, BabyAI, and TextWorld against RL, graph search, ReAct and Reflexion.
Problem
Go-Explore relies on hand-designed heuristics for selecting archived states, choosing exploratory actions, and deciding which states are interesting enough to archive, which are time-consuming to design and infeasible in complex domains.
Approach
Use a giant pretrained foundation model at all three Go-Explore stages (select a state from the archive, pick the next action, and judge whether a new state is interestingly new), keeping a state-conditional action history, so exploration heuristics come from the FM's notion of interestingness rather than hand-coding.
Key results
- On Game of 24, IGE reaches 100% success with GPT-4 by 150 operations, about 70.8% faster than the next best baseline (DFS).
- On TextWorld Coin Collector, IGE is the only method that finds the solution, while ReAct and Reflexion fail.
- On BabyAI Put Next To, IGE achieves 84% success versus 48% for ReAct, a statistically significant 36% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IGE instead of the strongest listed baseline, Reflexion | GPT-4 agents · Game of 24 | success rate (%): $83 \pm 10$ -> $\mathbf{100 \pm 0}$ | Helped |
| Replace uniform random action selection with intelligent action selection (ablation) | IGE · Game of 24 | success rate (%): \phantom{0}68 $\pm$ \phantom{0}9.0 -> \textbf{100 $\pm$ \phantom{0}0.0} | Helped |
| Replace uniform random action selection with intelligent action selection (ablation) | IGE · BabyAI (PN) | success rate (%): 24 $\pm$ 16 -> \textbf{84 $\pm$ 14} | Helped |
| Replace uniform random action selection with intelligent action selection (ablation) | IGE · TextWorld (CG) | success rate (%): \phantom{0}0 $\pm$ \phantom{0}0 -> \textbf{92 $\pm$ 10} | Helped |
| Use intelligent state selection instead of uniform random state selection (ablation) | IGE · BabyAI (PN) | success rate (%): 48 $\pm$ 20 -> \textbf{84 $\pm$ 14} | Helped |
| Use intelligent state selection instead of uniform random state selection (ablation) | IGE · TextWorld (CG) | success rate (%): 76 $\pm$ 16 -> \textbf{92 $\pm$ 10} | Helped |
| Use intelligent archive filtering instead of saving every discovered state (ablation) | IGE · BabyAI (PN) | success rate (%): 64 $\pm$ 20 -> \textbf{84 $\pm$ 14} | Helped |
| Use intelligent archive filtering instead of saving every discovered state (ablation) | IGE · TextWorld (CG) | success rate (%): 64 $\pm$ 20 -> \textbf{92 $\pm$ 10} | Helped |
| Maintain state-conditional action history (ablation) | IGE · Game of 24 | success rate (%): \phantom{0}33 $\pm$ \phantom{0}9.0 -> \textbf{100 $\pm$ \phantom{0}0.0} | Helped |
| Maintain state-conditional action history (ablation) | IGE · BabyAI (PN) | success rate (%): 72 $\pm$ 16 -> \textbf{84 $\pm$ 14} | Helped |
| Maintain state-conditional action history (ablation) | IGE · TextWorld (CG) | success rate (%): 72 $\pm$ 16 -> \textbf{92 $\pm$ 10} | Helped |
| Use domain-general prompts instead of original prompts (ablation) | IGE · Game of 24 | success rate (%): $100 \pm 0$ -> $96 \pm 4$ | Hurt |
Surprising findings
- Reflexion does not improve over ReAct in several games, which the authors attribute to overly long histories in long-horizon environments.
- Naive LLM and ReAct eventually plateau and are beaten by unintelligent DFS on Game of 24.
Limitations
- Archive comparisons against the entire archive at every new state may not scale to vast state spaces.
- Relies on the FM's judgments, which depend on model capability; weaker models show lower gains.
- Evaluated only on relatively small, text and gridworld-style environments.
Experimental setup
Datasets: Game of 24 test problems (ToT repo) BabyAI TextWorld Treasure Hunter, Cooking Game, Coin Collector
Benchmarks: Game of 24 BabyAI (text and visual) TextWorld Treasure Hunter TextWorld Cooking Game TextWorld Coin Collector
Metrics: success rate archive size
Baselines: Naive LLM ReAct Reflexion DFS BFS Classic Go-Explore GLAM Random action sampling Tree of Thoughts
Builds on: Go-Explore ReAct Reflexion Tree of Thoughts
Closest papers by meaning
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.88)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.86)
- First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs (NeurIPS 2024 · similarity 0.86)
- ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Search (ICLR 2024 · similarity 0.86)
- Dual-Scale World Models for LLM Agents towards Hard-Exploration Problems (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.