Research map / Multi-agent systems and games / LLM agents

CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CATArena, a tournament-based multi-round benchmark that measures code agents' evolutionary capability (slope of global performance across rounds) separately from single-turn base performance, across four competitive games and two objective optimization tasks.

Problem

Existing code agent benchmarks are static and single-turn, measuring pass rates or correctness and failing to assess iterative refinement, self-reflection, peer-learning, and code optimization over multiple rounds.

Approach

Model development as an iterative tournament: agents receive execution feedback (match logs, leaderboards, peer solutions) each round and refine code; a dual metric decouples base performance (round-1 relative strength) from evolutionary capability (OLS slope of global performance across rounds).

Key results

• Base performance and evolutionary capability are weakly correlated across tasks; e.g., Claude-4-Sonnet has S_base 0.90 on Chess but S_evo -0.061, while Gemini-2.5-Pro goes from 0.25 to +0.156 S_evo on Gomoku.
• Most agents show negligible or negative S_evo on Chess and SWE-Perf, and commercial agents do not show a decisive advantage in S_evo over the best minimal agent.
• In the peer-learning ablation, only Claude and Qwen effectively leverage peer code together with execution logs; most agents rely mainly on self-reflection.

Surprising findings

• Agents with high initial pass/base scores do not necessarily evolve well; low-start, high-growth trajectories occur (e.g., Gemini in Gomoku).
• Commercial code agents mostly improve static generation rather than evolutionary capability.

Limitations

• Long-term evolution run as a single chain due to cost, limiting statistical robustness; stability analysis only covers rounds 1-2.
• Evaluation horizon mostly limited to N<=4 rounds; performance saturates in later rounds.
• Some API tool-call failures and HTTP errors affect results, with default actions substituted.

Experimental setup

Datasets: OIBench SWE-Perf

Benchmarks: CATArena (Gomoku, Texas Hold'em, Bridge, Chess, OIBench, SWE-Perf)

Metrics: S_base S_evo (OLS slope of global performance) win rate Time AUC speedup ratio Pearson correlation of population dynamics

Baselines: Claude-4-Sonnet DeepSeek-3.1 Doubao-Seed-1.6 GPT-5 Gemini-2.5-Pro Qwen3-Coder-480B Claude Code Codex Gemini CLI Qwen Code

Builds on: OIBench SWE-Perf Google Agent Development Kit

Closest papers by meaning

• EvoMAS: Evolutionary Generation of Multi-Agent Systems (ICML 2026 · similarity 0.89)
• PACE: Two-Timescale Self-Evolution for Small Language Model Agents (NeurIPS 2026 · similarity 0.89)
• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.89)
• Simple Baselines are Competitive with Code Evolution (NeurIPS 2026 · similarity 0.88)
• From I/O to Code with Discovery Agent (NeurIPS 2026 · similarity 0.88)
• Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages (NeurIPS 2026 · similarity 0.88)