Research map / Multi-agent systems and games / Human-AI and goal-conditioned
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
BAGEL bootstraps LM web/tool agents without human supervision by iteratively relabeling randomly explored trajectories (LM labeler and zero-shot agent round-trips), then uses the resulting synthetic demonstrations as retrieved in-context examples.
Problem
LM agents executing natural-language instructions in digital environments (web browsers, tool APIs) generalize poorly to new environments without human demonstrations, which are costly to collect.
Approach
Shape random exploration trajectories into language-describable demonstrations by iterating round-trips between an LM trajectory labeler (trajectory to instruction) and a zero-shot LM agent (instruction to trajectory), filtered by an LM demonstration filter, then retrieve these demonstrations as in-context exemplars at test time; differs from HER by relabeling the full trajectory iteratively with
Key results
- On MiniWoB++ (hardest 10 tasks), trajectory-first BAGEL full reaches 61.0 vs 46.8 zero-shot, an improvement of over 13 points.
- On ToolQA, instruction-first full BAGEL reaches 43.3 F1 vs 40.9 zero-shot; trajectory-first full reaches 42.2.
- Execution failures on choose-date drop from 1.3 to 0.1 with BAGEL demonstrations, a reduction of up to 13x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add trajectory-first BAGEL demonstrations | PaLM-2 zero-shot agent · MiniWoB++ | average reward: 46.8 -> 61.0 | Helped |
| add instruction-first BAGEL demonstrations | PaLM-2 zero-shot agent · ToolQA | F1: 40.9 -> 43.3 | Helped |
| add trajectory-first BAGEL demonstrations | PaLM-2 zero-shot agent · ToolQA | F1: 40.9 -> 42.2 | Helped |
| add iterative re-labeling to trajectory-first BAGEL (ablation) | BAGEL · MiniWoB++ | average reward: 53.0 -> 61.0 | Helped |
| add iterative re-labeling to instruction-first BAGEL (ablation) | BAGEL · MiniWoB++ | average reward: 52.0 -> 56.0 | Helped |
| add iterative re-labeling to trajectory-first BAGEL (ablation) | BAGEL · ToolQA | F1: 40.9 -> 42.2 | Helped |
| add iterative re-labeling to instruction-first BAGEL (ablation) | BAGEL · ToolQA | F1: 38.8 -> 43.3 | Helped |
| use instruction-first rather than trajectory-first exploration (ablation) | BAGEL · MiniWoB++ | average reward: 61.0 -> 56.0 | Hurt |
| use instruction-first rather than trajectory-first exploration (ablation) | BAGEL · ToolQA | F1: 42.2 -> 43.3 | Helped |
| use random demonstrations instead of retrieving relevant demonstrations (ablation) | PaLM-2 agent on ToolQA · ToolQA | Accuracy: 42.2 -> 38.0 | Hurt |
| shuffle instruction-trajectory pairings in retrieved demonstrations (ablation) | PaLM-2 agent on ToolQA · ToolQA | Accuracy: 42.2 -> 41.4 | Hurt |
| add BAGEL demonstrations | PaLM-2 zero-shot agent · choose-date | average execution failures: 1.3 -> 0.1 | Helped |
Surprising findings
- Shuffled demonstrations (trajectories paired with random instructions) still improve over zero-shot, suggesting gains partly come from valid trajectory examples rather than instruction-conditional mapping.
- Instruction-first exploration underperforms trajectory-first on MiniWoB++, attributed to LM prior misalignment with the instruction distribution.
Limitations
- Gains on SciRex and search-engine are small or absent due to mismatch between synthetic and test instructions.
- Synthetic demonstrations skew toward certain modes and lack diversity (e.g., book-flight demos all in December).
- Relies on a closed PaLM-2 model and fixed hyperparameters; evaluated only in offline environments.
Experimental setup
Datasets: MiniWoB++ ToolQA
Benchmarks: MiniWoB++ ToolQA
Metrics: mean reward (mapped to [0,1]) F1 execution failures
Baselines: Zero-shot policy instruction-first no-iterations trajectory-first no-iterations Flat Zero-Shot SteP Zero-Shot RCI
Builds on: ReAct Hindsight Experience Replay PaLM-2
Closest papers by meaning
- Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025 · similarity 0.87)
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.87)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.87)
- Bootstrapping Cognitive Agents with a Large Language Model (AAAI 2024 · similarity 0.87)
- Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space (NeurIPS 2026 · similarity 0.86)
- Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.