Research map / Multi-agent systems and games / LLM agents
UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UltraHorizon, a benchmark of three exploration/rule-discovery environments that test LLM agents on long-horizon, partially observable tasks, finding frontier LLMs underperform humans, simple scaling fails, and proposing a context-refresh-with-notes-recall strategy.
Problem
Existing agent benchmarks are short-horizon and fully observable, leaving long-horizon, partially observable exploration and rule discovery under-evaluated.
Approach
Uses hidden-rule discovery across three interactive environments (Mystery Grid, Sequence Exploration, Alien Genetics Lab) with trajectories reaching 35k-200k+ tokens and 60-400+ tool calls, plus a failure taxonomy rooted in in-context locking and capability gaps.
Key results
- Gemini-2.5-Pro achieves the highest scores in Mystery Grid and Alien Genetics Lab among evaluated models, and all models underperform human participants.
- GLM-4.5 score on Mystery Grid falls monotonically from 34.4% at 1 hidden rule to 5.62% at 5 hidden rules.
- Naive step scaling for GLM-4.5 peaks at 125 steps (7.30) in Mystery Grid then declines, showing simple scaling fails.
Surprising findings
- Naive scaling of interaction steps does not reliably improve performance; scores peak then decline.
- Removing step limits lowered Qwen3-235b by 6.44 points in Sequence Exploration, suggesting overconfidence or premature giving up.
Limitations
- Evaluation relies on DeepSeek-R1 as LLM judge with point-wise scoring.
- Failure categorization is based on manual trajectory analysis.
- Only five LLMs evaluated and the environments are synthetic symbolic tasks.
Experimental setup
Benchmarks: UltraHorizon
Metrics: score (rule identification accuracy) tool calls trace length (tokens)
Baselines: Gemini-2.5-Pro GLM-4.5 DeepSeek-V3 Kimi K2-instruct Qwen3-235b-a22b-Instruct-2507 human participants
Builds on: needle-in-a-haystack
Closest papers by meaning
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
- SWE-Marathon: Can AI Agents Autonomously Complete Ultra-Long-Horizon Software Work? (NeurIPS 2026 · similarity 0.89)
- SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.89)
- Agentick: A Unified Benchmark for General Sequential Decision-Making Agents (NeurIPS 2026 · similarity 0.88)
- ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling (NeurIPS 2026 · similarity 0.88)
- ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.