Research map / Multi-agent systems and games / LLM agents
AGILE: A Novel Reinforcement Learning Framework of LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AGILE, an RL framework that trains an LLM agent (memory, tools, expert advice-seeking, reflection) end-to-end via SFT then PPO, and releases the ProductQA dataset; 7B/13B agents outperform GPT-4 agents on ProductQA, MedMCQA, and HotPotQA.
Problem
Complex conversational/QA tasks where LLM agents lack a unified, end-to-end optimizable framework combining memory, tools, reflection, and proactive seeking of expert advice; existing QA benchmarks don't test all these capabilities together.
Approach
Formulate the LLM agent as a token-level MDP where the LLM is the policy and executor functions (memory I/O, tool calls, expert advice, reflection) mutate state; train with imitation learning then session-level PPO so that advice-seeking decisions are learned via rewards that trade accuracy against a seeking-advice cost.
Key results
- On ProductQA, agile-vic13b-ppo reaches 0.854 short-answer accuracy and 0.784 total score, a 9.2% relative total-score gain over agile-gpt4-prompt (short answers).
- On MedMCQA, agile-mek7b-ppo reaches 85.2% accuracy with 31.6% advice rate, above gpt4-Medprompt's 79.1%.
- On HotPotQA, agile-vic13b-ppo reaches 67.5% exact-match accuracy vs 48.2% for the strongest baseline, with a 10.8% relative total-score gain over agile-gpt4-prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reflection (ablation) | AGILE with Vicuna-13b · ProductQA test | total score: 0.771 -> 0.784 | Helped |
| add memory (ablation) | AGILE with Vicuna-13b · ProductQA test | total score: 0.754 -> 0.784 | Helped |
| enable advice-seeking (ablation) | AGILE with Vicuna-13b · ProductQA test | accuracy: 0.747 -> 0.854 | Helped |
| make advice-seeking adaptive rather than forcing it in initial sessions (ablation) | AGILE with Vicuna-13b · ProductQA test | accuracy: 0.812 -> 0.854 | Helped |
| add tool use (ablation) | AGILE with Vicuna-13b · ProductQA test | total score: 0.717 -> 0.784 | Helped |
| add PPO training (ablation) | AGILE with Vicuna-13b · ProductQA test | total score: 0.766 -> 0.784; 2.3% | Helped |
| add reflection (ablation) | AGILE with Meerkat-7b · MedMCQA dev | accuracy: 0.790 -> 0.852 | Helped |
| add memory (ablation) | AGILE with Meerkat-7b · MedMCQA dev | accuracy: 0.741 -> 0.852 | Helped |
| enable advice-seeking (ablation) | AGILE with Meerkat-7b · MedMCQA dev | accuracy: 0.620 -> 0.852 | Helped |
| add PPO training (ablation) | AGILE with Meerkat-7b · MedMCQA dev | total score: 0.708 -> 0.726 | Helped |
| enable advice-seeking (ablation) | AGILE with Vicuna-13b · HotPotQA full dev | exact-match total score: 0.553 -> 0.628 | Helped |
| add PPO training (ablation) | AGILE with Vicuna-13b · HotPotQA full dev | exact-match total score: 0.617 -> 0.628 | Helped |
Surprising findings
- Random seeking-advice SFT initialization (agile-vic13b-ppo-random) outperformed the GPT-4-trajectory-initialized policy after PPO (0.798 vs 0.784 total score).
Limitations
- Expert advice is simulated by returning ground-truth answers, so real human-cost trade-offs are not tested.
- Long-answer accuracy relies on GPT-4 evaluator, validated on only 100 samples (94% agreement).
- The session-level advantage is a heuristic definition rather than a learned value function.
Experimental setup
Datasets: ProductQA MedMCQA HotPotQA Amazon Review Data
Benchmarks: ProductQA MedMCQA HotPotQA
Metrics: advice rate accuracy total score
Baselines: gpt3.5-prompt gpt4-prompt agile-gpt3.5-prompt agile-gpt4-prompt gpt4-Medprompt ReAct CRITIC Expel AutoAct
Builds on: PPO ReAct Vicuna Meerkat
Closest papers by meaning
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.88)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.88)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.87)
- From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory (ICML 2026 · similarity 0.87)
- Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space (NeurIPS 2026 · similarity 0.87)
- Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.