Research map / Multi-agent systems and games / LLM agents

LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LOCA-bench, an automatically scalable agentic benchmark (525 tasks, 8K–256K environment description length) that evaluates LLM agents and context-engineering scaffolds as context grows.

Problem

Existing long-context benchmarks test static single-step retrieval, not how language agents stay reliable as their context grows dynamically while exploring an environment ("context rot").

Approach

Keep task semantics fixed while programmatically scaling the initial environment state (via mock servers and templated generators) to control environment description length, and evaluate models combined with context-engineering scaffolds (tool-result clearing, compaction, memory tools, programmatic tool calling).

Key results

• Claude-4.5-Opus reaches 96.0% at 8K but drops to 14.7% at 256K; average 68.1%, the highest across models.
• At 128K, programmatic tool calling raises GPT-5.2-Medium from 38.7% to 49.3% accuracy and DeepSeek-V3.2-Thinking from 10.7% to 24.0%.
• Using the Claude Agent scaffold lowers Claude-4.5-Opus accuracy at 128K from 34.0% to 26.7%.

Surprising findings

• Claude Agent scaffold reduced Claude-4.5-Opus accuracy, as built-in subagents spawned without needed tools added irrelevant context.
• Context awareness hurt DeepSeek-V3.2-Thinking (10.7% to 4.0%) while helping Gemini-3-Flash and GPT-5.2-Medium.

Limitations

• Only 15 seed tasks and 525 samples; accuracy estimates on 75 samples per length are noisy.
• Own programmatic tool calling implementation underperforms Anthropic's official version, so scaffold comparisons depend on implementation details.

Experimental setup

Datasets: Toolathlon tasks (15 seed tasks)

Benchmarks: LOCA-bench

Metrics: accuracy (binary task success) trajectory length tool call count tool output length

Baselines: Claude-4.5-Opus GPT-5.2-Medium Gemini-3-Flash DeepSeek-V3.2-Thinking MiniMax-M2.1 GLM-4.7 Kimi-K2-Thinking Claude Agent

Builds on: Toolathlon GEM ReAct

Closest papers by meaning

• \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.90)
• AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.90)
• Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.90)
• Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.90)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.89)
• TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.89)