Research map / Multi-agent systems and games / LLM agents
Towards Self-Evolving Agent Benchmarks : Validatable Agent Trajectory via Test-Time Exploration
Dadi Guo, Tianyi Zhou, Dongrui Liu, Chen Qian, Qihan Ren, Shuai Shao, Zhiyuan Fan, Yi R. Fung, Kun Wang, Linfeng Zhang, Jing Shao
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-evolving agent benchmark framework that evolves task complexity via test-time exploration and validates agent trajectories to keep benchmarks from hitting ceilings.
Problem
Existing agent benchmarks are quickly saturated by newly developed agents, making it hard to evaluate agent capabilities.
Approach
A trajectory-based, reproducible benchmark complexity evolution method that uses test-time exploration to generate and validate agent trajectories so benchmark tasks can be evolved to harder complexity levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evolve GAIA tasks with TRACE (round 1) | GPT-5-Mini · GAIA total | Pass@1: 0.455 -> 0.260 | Hurt |
| evolve GAIA tasks with TRACE (round 2) | Gemini-2.5-Flash · GAIA Level 1 | Pass@1: 0.471 -> 0.040 | Hurt |
| evolve GAIA tasks with TRACE (round 2) | DeepSeek-V3.1 · GAIA total | Pass@1: 0.418 -> 0.188 | Hurt |
| evolve GAIA Level 3 tasks with TRACE | Kimi-K2, GPT-5-Mini · GAIA Level 3 | Pass@1: GPT-5-Mini 0.192 -> 0.375; DeepSeek-V3.1 0.231 -> 0.000 | Mixed |
| evolve GAIA with TRACE rounds | GPT-5-Mini · GAIA avg token length | tokens: 2864.5 -> 6275.7 | Improved |
| evolve AIME-2024 with TRACE (four rounds) | Qwen3-235B-A22B · AIME-2024 | Average accuracy: 0.9400 -> 0.7167 | Hurt |
| evolve AIME-2024 with TRACE (round 2) | DeepSeek-R1-Distill-Qwen-7B · AIME-2024 | Average length: 16890.5 -> 14853.4 | Mixed |
| add trajectory-agnostic solver as auxiliary validator (ablation) | TRACE · GAIA round 2 item count | items retained: 115 -> 69 | Hurt |
| evolve GAIA with TRACE | GAIA capability mix · Integrative capabilities share | percent: 4.0 -> 12.0 | Improved |
Closest papers by meaning
- SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents (NeurIPS 2025 · similarity 0.90)
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.89)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.89)
- A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks (NeurIPS 2026 · similarity 0.89)
- EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems (ICLR 2026 · similarity 0.89)
- MemEvolve: Meta-Evolution of Agent Memory Systems (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
