Research map / Multi-agent systems and games / LLM agents

Self-evolving LLM agents with in-distribution Optimization

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-Evolve, a self-evolving LLM agent framework that learns a weighted IQL critic on a hybrid expert and self-collected dataset, derives process rewards via GAE, and updates the policy with behavior-proximal policy optimization.

Problem

Long-horizon LLM agents receive sparse, delayed episodic rewards, making credit assignment hard; existing process reward methods are distribution-sensitive and need online search or extensive interaction.

Approach

Learn an in-distribution critic (weighted IQL) on a fixed hybrid offline dataset, derive step-wise process rewards from GAE advantages (excluding auxiliary rule-based rewards), and update the policy with an asymmetric-clipped behavior-proximal objective, iterating the data collection loop.

Key results

• On ALFWorld with Llama-2-7B-Chat, Q-Evolve reaches 90.7 seen / 89.6 unseen versus QLASS 77.9 / 82.8.
• Average across WebShop, SciWorld, ALFWorld is 79.4 for Q-Evolve versus 74.5 for QLASS (best baseline).
• On Qwen2.5-7B-Instruct ALFWorld, Q-Evolve (1-iter) uses 13K env steps and gets 88.6 seen / 87.3 unseen, versus 320K steps for online RL baselines.

Surprising findings

• Using the critic for test-time reranking (w/o PI) underperforms the behavior cloning policy, attributed to distribution mismatch.
• Including the auxiliary retrospective reward inside GAE hurts performance relative to using only environment reward.

Limitations

• Retrospective reward design is task-specific and relies on structured environment feedback.
• Greedy rollouts reduce trajectory diversity across iterations.
• Distribution shift accumulates across iterations and is not explicitly corrected.

Experimental setup

Datasets: ALFWorld WebShop ScienceWorld

Benchmarks: WebShop ScienceWorld ALFWorld

Metrics: average accumulated reward success rate env steps

Baselines: GPT-4 GPT-3.5-Turbo Reflexion SFT RFT PPO Best-of-N ETO DMPO QLASS KnowAgent WKM

Builds on: Implicit Q-Learning Generalized Advantage Estimation PPO ETO QLASS

Closest papers by meaning

• StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models (ICLR 2026 · similarity 0.90)
• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.89)
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.89)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.89)
• CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards (ICLR 2026 · similarity 0.89)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.89)