Research map / Multi-agent systems and games / LLM agents

$\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Multi^2, a hierarchical two-agent LLM framework where a SFT-trained high-level planner (System 1) emits sub-goals and a low-level executor (System 2) is trained with offline-to-online actor-critic RL, evaluated on ScienceWorld, ALFWorld, and TextCraft.

Problem

Long-horizon interactive LLM agents are brittle: they suffer objective drift as goals and plans drift over extended interaction, and they are token-inefficient due to growing histories.

Approach

Decouple task decomposition (System 1, SFT sub-goal planner) from atomic action execution (System 2, offline-to-online RL executor) with separate LoRA adapters, using a policy-anchored advantage term in offline training and a KL-regularized advantage-weighted online update.

Key results

• On ScienceWorld ID with Llama-3.1 8B, Multi^2 reaches 67.61% vs. Glider 60.48% and ReAct 23.23%.
• On ALFWorld OOD with Qwen-2.5 3B, Multi^2 reaches 49.29% vs. Glider 41.43%.
• On TextCraft with Mistral 7B, Multi^2 reaches 44.50% success vs. Glider 28.50% and ADaPT 30.20%.

Surprising findings

• On ScienceWorld OOD with Llama-3.1 8B, Glider (34.36%) slightly outperforms Multi^2 (30.68%).
• RL-only training of the high-level planner yields low median and high variance, and swapped RL/SFT assignment performs worst.

Limitations

• Datasets are built from GPT-4-Turbo trajectories released by AgentGym, so quality depends on that teacher data.
• Evaluated only on three text-based interactive benchmarks; math and code benchmarks were explicitly excluded.
• Gains are not uniform: Glider is slightly better on ScienceWorld OOD with Llama-3.1 8B.

Experimental setup

Datasets: AgentGym GPT-4-Turbo trajectories ScienceWorld ALFWorld TextCraft

Benchmarks: ScienceWorld ALFWorld TextCraft

Metrics: pass@1 score/success rate (ID and OOD) token efficiency (performance/tokens)

Baselines: ReAct Reflexion ADaPT GRPO Glider

Builds on: Glider ReAct ADaPT IQL AWAC

Closest papers by meaning

• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.91)
• Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning (ICML 2025 · similarity 0.89)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.89)
• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.89)
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.89)
• Can large language models explore in-context? (NeurIPS 2024 · similarity 0.89)