Research map / Multi-agent systems and games / LLM agents

Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Gaia2, an asynchronous, event-driven benchmark (1,120 scenarios in a smartphone-like Mobile environment) with a write-action verifier usable for RLVR, released alongside the ARE platform, and evaluates frontier and open LLM agents.

Problem

Existing LLM agent benchmarks are static or synchronous: environments change only when the agent acts, so asynchronous events, temporal constraints, noise, ambiguity, and multi-agent collaboration are untested.

Approach

Environments evolve independently of the agent via event DAGs and time advancement, with a verifier that checks every state-changing write action against oracle annotations, enabling action-level evaluation and RL reward signals.

Key results

• GPT-5 (high) achieves the highest overall pass@1 of 42.1% on Gaia2, with Claude-4 Sonnet Thinking at 37.8% and Claude-4 Sonnet at 34.8%.
• Kimi-K2 is the best open-source model at 20.1% overall pass@1.
• The ARE Verifier reaches 0.98 agreement, 0.99 precision and 0.95 recall against 450 hand-labeled trajectories, versus 0.72 agreement for an LLM-only judge.

Surprising findings

• GPT-5 (high) scores 0.0% on the Time split, while reasoning-heavy models are slower and miss deadlines.
• Heterogeneous A2A teams with a strong app-agent improve pass@1 over the all-light team, and PTC improves efficiency but not performance.

Limitations

• Current universe generation leaves several cross-app temporal and semantic consistency dependencies unhandled.
• Single-threaded sequential scaffold cannot express some concurrent time-window scenarios.
• Annotation interface is not released, and verifier relies on an LLM rubric for flexible text arguments.

Experimental setup

Datasets: PersonaHub Amazon product dataset Wikipedia data

Benchmarks: Gaia2

Metrics: pass@1 cost per scenario time to solve agreement precision recall

Baselines: GPT-5 Claude-4 Sonnet Gemini-2.5-Pro Grok-4 Kimi-K2 Qwen3-235B GPT-OSS 120B Llama 4 Maverick Llama 3.3 70B Instruct GPT-4o

Builds on: ReAct GAIA ToolSandbox AppWorld tau-bench

Closest papers by meaning

• PACE: Two-Timescale Self-Evolution for Small Language Model Agents (NeurIPS 2026 · similarity 0.89)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition (ICML 2026 · similarity 0.89)
• Test-Time Adaptation for LLM Agents via Environment Interaction (ICLR 2026 · similarity 0.89)
• AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.89)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.88)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.88)