Research map / Large language models / Reasoning and chain of thought

Evaluating Test-Time Scaling of General LLM Agents

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces General AgentBench, a unified multi-domain agent benchmark, and studies sequential and parallel test-time scaling of ten LLM agents, finding a scaling plateau and a verification gap that limit gains.

Problem

Test-time scaling behavior of general-purpose LLM agents in realistic settings where the task domain is not specified and tools span multiple domains is poorly understood; existing agent benchmarks are domain-specific.

Approach

A unified MCP-based benchmark exposing search, coding, reasoning and tool-use tools in one shared registry, plus a statistically grounded paired-bootstrap definition of a "scaling plateau" for sequential scaling and a pass@K vs. commitment-accuracy analysis (the "verification gap") for parallel scaling.

Key results

• Most models degrade substantially moving from domain-specific to General AgentBench (average drops ~10-30%), with Claude Sonnet 4.5 showing only a 7.0% average drop and GPT-5 dropping 22.7% (60.9 to 47.1).
• Sequential scaling shows a plateau: e.g., Qwen3-235B on Search has a statistically significant gain up to 112k context budget, with the lower bound turning negative at 112k to 128k (C_4 = -0.045).
• Parallel scaling raises pass@K by roughly 50% from K=1 to K=4, but self-choice and majority voting consistently lag behind pass@K, revealing a verification gap.

Surprising findings

• GPT-5 as an external verifier generally underperforms models' own self-judgment in parallel selection, attributed to solution familiarity.
• Several models gain on search tasks in the general setting via cross-domain use of MCP tools, despite the overall degradation.

Limitations

• Benchmarks use subsampled instances (e.g., 50-124 per dataset), limiting statistical power for some domains.
• Results depend on API-accessed models and a single judge (GPT-5) for external verification.
• Sequential scaling strategies beyond continue-thinking were only tested on two models and two domains.

Experimental setup

Datasets: BrowseComp WebVoyager SWE-Bench Verified Terminal-Bench MathHay Tau2-Bench MCP-Bench LongBench v2 HELMET MRCR

Benchmarks: General AgentBench BrowseComp WebVoyager SWE-Bench Verified Terminal-Bench MathHay Tau2-Bench MCP-Bench LongBench HELMET MRCR

Metrics: accuracy pass@K relative change bootstrap confidence interval paired-bootstrap lower bound

Baselines: Domain-specific evaluation (baseline setting) Plain web search agent Majority voting Pairwise comparison Self-choice

Builds on: Model Context Protocol BrowseComp SWE-Bench Verified Terminal-Bench MathHay Tau2-Bench MCP-Bench WebVoyager

Closest papers by meaning

• LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (NeurIPS 2026 · similarity 0.90)
• ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management (ICML 2026 · similarity 0.90)
• Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute (AAAI 2026 · similarity 0.89)
• AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.89)
• ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling (NeurIPS 2026 · similarity 0.88)
• Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling (NeurIPS 2025 · similarity 0.88)