Research map / Large language models / Agents, code and math
ExpeL: LLM Agents Are Experiential Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExpeL, a prompt-only LLM agent that gathers trial-and-error experiences on training tasks, distills cross-task natural-language insights, and retrieves successful trajectories as in-context examples, improving gpt-3.5-turbo ReAct agents on HotpotQA, ALFWorld, WebShop and transferring insights to FEVER without parameter updates.
Problem
LLM agents for decision-making are either finetuned (costly, needs weights, may hurt generalization) or prompted (no memory across tasks, so no learning from experience beyond a few demonstrations). The paper asks how an agent can learn from experience across tasks without parametric updates, e.g.,
Approach
An agent gathers success and failure trajectories via Reflexion-style retries on training tasks, stores them in an experience pool, and uses an stronger LLM (gpt-4) to extract natural-language insights through ADD/EDIT/UPVOTE/DOWNVOTE operations with importance counts. At test time it combines the insight list with top-k similar successful trajectories retrieved by embedding similarity as in-conte
Key results
- ExpeL reaches 39.0% SR on HotpotQA vs 28.0% for ReAct, and 59.0% SR on ALFWorld vs 40.0% for ReAct (ablation table).
- On ALFWorld, ExpeL with one attempt reaches 54% vs Reflexion's 59% at R3 per the main text claim, while ExpeL+Reflexion reaches 64.2% at R3 (Table 2).
- Transferring HotpotQA insights to FEVER raises success from 63% (ReAct) to 70% with task-demo finetuned insights.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adapted HotpotQA insights using target-task demonstrations | ExpeL Transfer · FEVER | success rate (%): 63 -> \textbf{70} | Helped |
| Add target-task demonstrations when adapting transferred insights (ablation) | ExpeL Transfer · FEVER | success rate (%): 65 -> \textbf{70} | Helped |
| Add Reflexion to ExpeL | ExpeL · ALFWorld | success rate (%): 54.5\% -> \textbf{\hl{59.0\%}} | Helped |
| Replace LLM-generated insights with hand-crafted insights (ablation) | ExpeL · HotpotQA | success rate (%): \textbf{39.0} -> 32.0 | Hurt |
| Add reflections to insight construction (ablation) | ExpeL · HotpotQA | success rate (%): \textbf{39.0} -> 29.0 | Hurt |
| Use gpt-3.5-turbo instead of gpt-4 for insight extraction (ablation) | ExpeL · HotpotQA | success rate (%): \textbf{39.0} -> 32.0 | Hurt |
| Rank retrieved trajectories by reasoning similarity instead of task similarity (ablation) | ExpeL · ALFWorld | success rate (%): \textbf{59.0} -> 48.5 | Hurt |
| Randomly sample successful trajectories instead of ranking by task similarity (ablation) | ExpeL · ALFWorld | success rate (%): \textbf{59.0} -> 42.5 | Hurt |
| Gather diverse success and failure experiences using Reflexion rather than only ReAct (ablation) | ExpeL experience gathering · HotpotQA | success rate | Helped |
| Gather autonomous experiences rather than extract insights only from initial fewshot examples (ablation) | ExpeL experience gathering · HotpotQA | success rate | Helped |
Surprising findings
- Insights extracted from fewshot examples only (no gathered experience) gave no advantage over ReAct on HotpotQA, showing collected experience is essential.
- Adding Reflexion reflections into insight extraction hurt HotpotQA performance (29.0% vs 39.0%).
Limitations
- Evaluated only on text-based observations; no image observations.
- Relies on closed-source API LLMs; no open-source model experiments.
- Prompting lacks theoretical guarantees; insight list may eventually exceed context window for lifelong learning.
Experimental setup
Datasets: HotpotQA ALFWorld WebShop FEVER
Benchmarks: HotpotQA ALFWorld WebShop FEVER
Metrics: success rate mean reward
Baselines: ReAct Act Reflexion Imitation learning
Builds on: ReAct Reflexion Faiss
Closest papers by meaning
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.89)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.89)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.88)
- Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.88)
- Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025 · similarity 0.88)
- ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.