Research map / Large language models / Agents, code and math
Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a ReAct-style LLM agent can improve by accumulating its own successful trajectories as retrieved in-context examples, further boosted by population-based database curation and exemplar-level utility-based curation.
Problem
LLM agents for sequential decision-making usually need heavy task-specific knowledge engineering (prompts, human-curated examples, custom observation/action spaces); the paper asks whether agents can improve automatically by curating a database of their own successful trajectories used as in-context
Approach
Bootstrap a trajectory database from the agent's own successful training-task attempts, retrieved per decision point via multi-key KNN; add population-based training over N parallel databases (TrajBootstrap-DBCuration) and a retrieval-weighted quality metric to keep high-utility exemplars (ExemplarCuration).
Key results
- TrajBootstrap raises success from 0.73 to 0.89 on ALFWorld, 0.75 to 0.79 on InterCode-SQL, and 0.55 to 0.64 on Wordcraft with gpt-4o-mini.
- TrajBootstrap+DBCuration+ExemplarCuration reaches 0.93 on ALFWorld, surpassing AutoManual (0.91) that uses gpt-4-turbo plus gpt-4o-mini with hand-crafted action/observation spaces.
- On ALFWorld, the trajectory bootstrapping gain (15 to 20 points) is comparable to upgrading Fixed-DB from gpt-4o-mini to gpt-4o (0.88).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Accumulate self-generated successful trajectories instead of using a fixed database | GPT-4o-mini ReAct agent · ALFWorld | success rate: 0.73±0.02 -> 0.89±0.01 | Helped |
| Accumulate self-generated successful trajectories instead of using a fixed database | GPT-4o-mini ReAct agent · InterCode-SQL | success rate: 0.75±0.01 -> 0.79±0.01 | Helped |
| Accumulate self-generated successful trajectories instead of using a fixed database | GPT-4o-mini ReAct agent · Wordcraft | success rate: 0.55±0.03 -> 0.64±0.03 | Helped |
| Add population-based database curation | Trajectory bootstrapping · ALFWorld | success rate: 0.89±0.01 -> 0.91±0.01 | Helped |
| Add population-based database curation | Trajectory bootstrapping · InterCode-SQL | success rate: 0.79±0.01 -> 0.78±0.01 | Hurt |
| Add population-based database curation | Trajectory bootstrapping · Wordcraft | success rate: 0.64±0.03 -> 0.64±0.01 | No clear effect |
| Select high-utility successful trajectories as exemplars | Trajectory bootstrapping · ALFWorld | success rate: 0.89±0.01 -> 0.90±0.02 | No clear effect |
| Select high-utility successful trajectories as exemplars | Trajectory bootstrapping · InterCode-SQL | success rate: 0.79±0.01 -> 0.81±0.01 | Helped |
| Select high-utility successful trajectories as exemplars | Trajectory bootstrapping · Wordcraft | success rate: 0.64±0.03 -> 0.72±0.02 | Helped |
| Combine database-level and exemplar-level curation | Trajectory bootstrapping · ALFWorld | success rate: 0.89±0.01 -> 0.93±0.03 | Helped |
| Combine database-level and exemplar-level curation | Trajectory bootstrapping · InterCode-SQL | success rate: 0.79±0.01 -> 0.82±0.01 | Helped |
| Combine database-level and exemplar-level curation | Trajectory bootstrapping · Wordcraft | success rate: 0.72±0.02 -> 0.69±0.01 | Hurt |
Surprising findings
- Naive accumulation of self-generated successful trajectories matches the gain of roughly two to three test-time attempts (pass@2 to pass@3).
- Database-level curation did not improve Wordcraft performance, and combining it with exemplar curation lowered Wordcraft success relative to exemplar curation alone.
Limitations
- Relies on the in-context capabilities of the base LLM; effectiveness may vary with weaker models.
- Performance is not monotonic in database size and shows variance across trials and within trials.
- Only successful trajectories are stored; failed trajectories are left to future work due to credit assignment difficulty.
Experimental setup
Datasets: ALFWorld InterCode-SQL Wordcraft
Benchmarks: ALFWorld InterCode-SQL Wordcraft
Metrics: success rate pass@k
Baselines: Fixed-DB AutoGuide AutoManual GameSQL GameSQL+Cheat gpt-4o Fixed-DB Fixed-DB pass@k
Builds on: ReAct AutoGuide AutoManual ExpeL RAP
Closest papers by meaning
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.89)
- AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents (NeurIPS 2024 · similarity 0.89)
- Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments (ICLR 2025 · similarity 0.89)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.89)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.89)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.