Research map / Multi-agent systems and games / LLM agents

Credit-Budgeted ICPC-Style Coding: When LLM Agents Must Pay for Every Decision

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces USACOArena, an ICPC-style interactive arena where coding agents pay a fixed credit budget for LLM tokens, tests, hints, and elapsed time, and evaluates frontier LLM agents and Codex swarms on USACO 2024-2025 problems.

Problem

Existing coding-agent benchmarks measure only final correctness under implicitly infinite resources, ignoring the token, test, and time costs that real agents incur.

Approach

A credit economy in which every LLM call, local test, hint, and second of wall-clock time depletes a fixed budget, with ICPC-style all-or-nothing scoring, turning a static coding test into a budget-constrained competition over USACO contests.

Key results

• Gemini-2.5-pro achieved the top average rank (1.30) with average score 14.00 and consumed about 13.8M credits, while GPT-5-Codex ranked second with 9.39 average score.
• Reducing the credit limit to 10M dropped Gemini-2.5-pro from 13.2 to 8.3, but doubling it to 40M gave no gain (13.0), which the authors read as capability-limited rather than budget-limited.
• In Codex swarm profiling, the Speedy Spendthrift scored 3 due to communication overhead, while the Cost-Aware Strategist reached 8.

Surprising findings

• Doubling the credit budget to 40M did not improve the top model's score, suggesting current agents are capability-limited rather than budget-limited.
• Gemini-2.5-pro's broad-exploration strategy outperformed the more precise but conservative GPT-5-Codex in the ranking.

Limitations

• Ranking depends on a self-defined credit economy and time coefficient alpha, and the main experiments set alpha=0 to avoid network latency confounds.
• Small problem corpus (48 problems over one season) with 5 runs per agent, and the ablations use only 3 runs due to compute constraints.
• Swarm experiments are limited to Codex agents and a small set of profiles.

Experimental setup

Datasets: USACO 2024-2025 season problems (48 problems)

Benchmarks: USACOArena (USACO 2024-2025 contests)

Metrics: score consumed credit rank accepted rate

Baselines: Gemini-2.5-pro GPT-5-Codex Claude-4-Sonnet Qwen3-235B GLM-4.5 DeepSeek-V3.1 DeepSeek-V3 Kimi-K2

Builds on: ACM-ICPC competition format USACO Model Context Protocol

Closest papers by meaning

• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.87)
• Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve (NeurIPS 2025 · similarity 0.87)
• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.87)
• CORVUS: Context Optimization and Reduction Via Underlying Synchronization for LLM Coding Agents (NeurIPS 2026 · similarity 0.87)
• VeRO: A Harness for Agents to Optimize Agents (ICML 2026 · similarity 0.86)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.86)