Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Dr. Strategy: Model-Based Generalist Agents with Strategic Dreaming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Dr. Strategy is a model-based RL agent that learns discrete latent landmarks via VQ-VAE and uses a landmark-conditioned highway policy plus strategic exploration and focused-sampling achievement to dream in a divide-and-conquer manner for pixel-based navigation.
Problem
Pixel-based model-based generalist agents dream with naive strategies (random sampling from the replay buffer), which is sample-inefficient in large, partially observable, visually complex navigation tasks.
Approach
Learn latent landmarks (VQ-VAE codes over world-model states) and a landmark-conditioned highway policy; select a curious landmark for exploration and the nearest landmark to a goal for achievement, and train the achiever with focused sampling of nearby state pairs.
Key results
- Dr. Strategy reaches 94.03% success on 9-room, 96.50% on spiral 9-room, and 67.11% on 25-room, versus 19.75%/21.19%/9.62% for LEXA.
- On Maze-7x7 Dr. Strategy achieves 86.66% vs 51.11% for LEXA; on Maze-15x15 44.44% vs 41.20%.
- On RoboKitchen Dr. Strategy gets 19.44%, below LEXA's 24.30%, i.e., comparable rather than superior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Dr. Strategy's strategic dreaming instead of GC-Director | zero-shot goal achievement · 9-Room | success rate: 28.08% -> 94.03% | Helped |
| Use Dr. Strategy's strategic dreaming instead of GC-Director | zero-shot goal achievement · Spiral 9-Room | success rate: 30.45% -> 96.50% | Helped |
| Use Dr. Strategy's strategic dreaming instead of GC-Director | zero-shot goal achievement · 25-Room | success rate: 27.11% -> 67.11% | Helped |
| Use Dr. Strategy's strategic dreaming instead of LEXA | zero-shot goal achievement · Maze-7x7 | success rate: 51.11% -> 86.66% | Helped |
| Use Dr. Strategy's strategic dreaming instead of LEXA | zero-shot goal achievement · Maze-15x15 | success rate: 41.20% -> 44.44% | Helped |
| Use Dr. Strategy instead of LEXA | zero-shot goal achievement · RoboKitchen | success rate: 24.30% -> 19.44% | Hurt |
| Add strategic exploration using curious landmarks and the Highway policy (ablation) | Dr. Strategy · unseen-goal evaluation environments | success rate | Helped |
| Add strategic achievement using landmarks and the Highway policy (ablation) | Dr. Strategy · unseen-goal evaluation environments | success rate | Helped |
| Train the Achiever with focused sampling of nearby start and goal states (ablation) | Dr. Strategy · unseen-goal evaluation environments | success rate | Helped |
| Increase the number of landmarks from 16 to 64 (ablation) | Dr. Strategy · 9-Room | success rate | Mixed |
| Increase the number of landmarks to 512 (ablation) | Dr. Strategy · 9-Room and Maze-7x7 | success rate | Hurt |
| Use 64 landmarks for 3D-Maze navigation (ablation) | Dr. Strategy · 3D-Maze | success rate | Helped |
Surprising findings
- Removing focused sampling from the achiever drops performance to roughly the level of the ablation without strategic achievement, indicating the nearby-goal training is critical.
- GC-Director, a hierarchical method, performs very poorly on RoboKitchen (1.45%).
Limitations
- Limited performance on the robotic manipulation RoboKitchen environment.
- Number of landmarks is a fixed hyperparameter; performance gap shrinks in Maze-15x15 due to visually repeated regions confusing the highway policy.
Experimental setup
Datasets: 2D navigation (9-room, spiral 9-room, 25-room) Maze-7x7 and Maze-15x15 (modified Memory Maze) RoboKitchen
Benchmarks: 2D Navigation 9-room 2D Navigation spiral 9-room 2D Navigation 25-room 3D-Maze Maze-7x7 3D-Maze Maze-15x15 RoboKitchen
Metrics: zero-shot goal success rate
Baselines: LEXA LEXA-Explore GC-Director PEG
Builds on: LEXA DreamerV2 Director Choreographer Memory Maze
Closest papers by meaning
- Mastering Memory Tasks with World Models (ICLR 2024 · similarity 0.85)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.85)
- StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction (NeurIPS 2026 · similarity 0.84)
- Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models (ICLR 2025 · similarity 0.84)
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.83)
- Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.