Research map / Large language models / Agents, code and math
Reason for Future, Act for Now: A Principled Architecture for Autonomous LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAFA, a prompting-based LLM agent architecture that alternates in-context learning of a model/critic from a memory buffer with tree-search planning and executes only the first action, with a Bayesian regret analysis giving sqrt(T) bounds.
Problem
Autonomous LLM agents in interactive decision-making need to complete tasks with few environment interactions, but LLM reasoning is stateless and ungrounded, and there is no principled RL-style framework or provable sample-efficiency guarantee for LLM reasoning-plus-acting.
Approach
Casts LLM reasoning as Bayesian adaptive MDP learning and planning: the LLM is prompted with a growing memory buffer to act as a model and critic (in-context posterior/BMA estimate), plans a multi-step trajectory via prompted tree search, executes only the first action, and replans, with an entropy-based switching condition giving a sqrt(T) Bayesian regret bound.
Key results
- On Game of 24 with gpt-4, RAFA (B=2) reaches 93% success vs. 81% for ToT (B=2) and 21% for Reflexion.
- On ALFWorld with gpt-3, RAFA achieves 99.25% overall success vs. 92.54% for Reflexion and 61.94% for ReAct.
- Theory: a Bayesian regret of order sqrt(T) (up to log factors, e.g. O~(sqrt(d^3 T)) for d-dimensional Bayesian linear kernel MDPs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAFA instead of Reflexion | GPT-4 agent · Game of 24 | success rate: 21% -> 89% | Helped |
| Use RAFA instead of ToT | GPT-3.5 agent · Game of 24 | success rate: 17% -> 46% | Helped |
| Increase search breadth from B=1 to B=2 (ablation) | RAFA with GPT-4 · Game of 24 | success rate: 89% -> 93% | Helped |
| Increase search breadth from B=1 to B=2 (ablation) | RAFA with GPT-3.5 · Game of 24 | success rate: 29% -> 46% | Helped |
| Use RAFA instead of Reflexion | GPT-3 agent · ALFWorld Total | success rate: 92.54 -> 99.25 | Helped |
| Extend the interaction horizon from T=1 to T=7 (ablation) | RAFA playing O against GPT-4 · Tic-Tac-Toe | O winning rate: 10% -> 100% | Helped |
| Extend the interaction horizon from T=1 to T=5 (ablation) | RAFA playing O against GPT-4 · Tic-Tac-Toe | O winning rate: 10% -> 50% | Helped |
Surprising findings
- On Game of 24 with gpt-3.5, RAFA's gains are large (46% vs. 17% for ToT B=2), but the paper notes that LLMs predict more accurately with more in-context feedback only on some tasks shown.
- AdaPlanner outperforms RAFA at the very first ALFWorld episode, attributed to handcrafted rejection programs.
Limitations
- Practical implementations of the optimistic-bonus and posterior-sampling variants are not provided.
- Theoretical guarantees rely on an posterior-alignment assumption for LLMs and a regularity assumption on MDPs.
- Experiments use mainly text-based benchmarks and small task suites.
Experimental setup
Datasets: ALFWorld Game of 24 BlocksWorld
Benchmarks: Game of 24 ALFWorld BlocksWorld Tic-Tac-Toe
Metrics: success rate winning rate sample efficiency (success vs. episode/step) value estimation accuracy
Baselines: ReAct Reflexion AdaPlanner BUTLER Tree of Thoughts (ToT) RAP CoT
Builds on: ReAct Reflexion Tree of Thoughts Reasoning via Planning AdaPlanner
Closest papers by meaning
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.90)
- Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning (ICML 2026 · similarity 0.89)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.88)
- Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.88)
- Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making (NeurIPS 2026 · similarity 0.87)
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.