Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Behavioral Exploration: Learning to Explore via In-Context Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes behavioral exploration (BE), a transformer diffusion policy trained by behavioral cloning conditioned on history and a coverage-to-go value, enabling in-context online exploration over expert demonstration behaviors.
Problem
Fast online exploration and adaptation for autonomous agents: existing methods rely on random exploration and slow gradient-based updates, and offline RL-style exploration methods need reward labels.
Approach
Train a long-context diffusion policy with standard behavioral cloning, additionally conditioning on the history of visited states and on a coverage-to-go value (trace of inverse feature covariance over the future plus history). At deployment, feeding past states into the context and setting a high coverage value induces in-context adaptive exploration over expert-like behaviors.
Key results
- On D4RL Antmaze, BE reaches 0.556 final goal success vs 0.285 for BC and 0.195 for SUPE, and 24.1 vs 20.1 regions for BC.
- On Libero hidden-task evaluation, BE achieves 0.824 success vs 0.590 for BC, with 2-3x fewer attempts than BC.
- On WidowX real-robot reaching, BE succeeds roughly 40% more trials than BC or OpenVLA and needs about 2 fewer attempts on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with behavioral exploration rather than standard BC | D4RL Antmaze · D4RL Antmaze medium & large | goals reached: 0.285 { $\pm$ 0.015} -> 0.556 { $\pm$ 0.013} | Helped |
| train with behavioral exploration rather than standard BC | D4RL Antmaze · D4RL Antmaze medium & large | regions reached: 20.056 { $\pm$ 0.150} -> 24.118 { $\pm$ 0.253} | Helped |
| train with behavioral exploration rather than SUPE | D4RL Kitchen · D4RL Kitchen | tasks completed: 0.956 { $\pm$ 0.011} -> 0.922 { $\pm$ 0.014} | Hurt |
| train with behavioral exploration rather than standard BC | Libero · Libero task hidden | success rate: 0.590 { $\pm$ 0.013} -> 0.824 { $\pm$ 0.010} | Helped |
| add Gaussian action noise to BC (ablation) | BC on Libero · Libero task hidden | success rate: 0.590 { $\pm$ 0.013} -> 0.684 { $\pm$ 0.013} | Helped |
| add Gaussian action noise to BC (ablation) | BC on Libero · Libero task provided | success rate: 0.977 { $\pm$ 0.004} -> 0.950 { $\pm$ 0.006} | Hurt |
| condition BC on observation history (ablation) | BC on Libero · Libero task hidden | success rate: 0.590 { $\pm$ 0.013} -> 0.643 { $\pm$ 0.013} | Helped |
| condition BC on observation history (ablation) | BC on Libero · Libero task provided | success rate: 0.977 { $\pm$ 0.004} -> 0.804 { $\pm$ 0.011} | Hurt |
| train with behavioral exploration rather than standard BC | Libero · Libero task provided | success rate: 0.977 { $\pm$ 0.004} -> 0.993 { $\pm$ 0.002} | Helped |
| add “other” to the silver-pot prompt (ablation) | OpenVLA · WidowX Task 1 | left contacts: 1/8 -> 2/8 | Helped |
| add “other” to the silver-pot prompt (ablation) | OpenVLA · WidowX Task 1 | neither-object contacts: 0/8 -> 2/8 | Hurt |
Surprising findings
- Naively conditioning BC on history can significantly hurt performance, while BE's history conditioning improves it.
- Action noise added to BC improves hidden-task success but significantly hurts performance when the task is provided.
Limitations
- BE is limited to exploring behaviors present in the demonstration data and may not explore beyond it.
- Real-world evaluation is limited to a small set of reaching tasks with 8 trials per task and method.
- The OpenVLA comparison uses a model with about 100x more parameters trained on a larger dataset, making the comparison not like-for-like.
Experimental setup
Datasets: D4RL Antmaze diverse D4RL Kitchen partial Libero 90 BridgeData V2
Benchmarks: D4RL Antmaze medium and large D4RL Kitchen Libero 90 WidowX robot reaching
Metrics: percentage of goals reached number of regions reached percentage of tasks completed task success rate number of attempts to success
Baselines: BC Online RND ExPLORe HILP SUPE BC with action noise BC with history conditioning OpenVLA
Builds on: diffusion policy RND HILP SUPE ExPLORe
Closest papers by meaning
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.89)
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.89)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.88)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.88)
- SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.