Research map / Large language models / Agents, code and math
CodeClash: Benchmarking Goal-Oriented Software Engineering
John Yang, Kilian Lieret, Joyce Yang, Carlos E. Jimenez, Muhtasham Oblokulov, Aryan Siddiqui, Ofir Press, Ludwig Schmidt, Diyi Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| show players a read-only copy of opponent's previous code (ablation) | GPT-5 · Core War | round win rate: +7.8% (74.6%) | Improved |
| show players a read-only copy of opponent's previous code (ablation) | Claude Sonnet 4.5 · Core War | round win rate: -1.8% (53.2%) | No clear effect |
| show players a read-only copy of opponent's previous code (ablation) | Gemini 2.5 Pro · Core War | round win rate: -5.5% (22.7%) | Hurt |
| increase from 2 to 6 players per tournament (ablation) | Core War tournaments · Core War | lead changes: 18.2% -> 48.4% | Improved |
| replace mini-SWE-agent with SWE-agent scaffold (ablation) | three models · CC:Ladder | ladder rank: at most 2 ranks change | No clear effect |
| randomly shuffle player order (ablation) | MIT Battlecode 2025 arena · dummy-player tournaments | win rate: positional advantage -> 50% win rate | Improved |
| lose several consecutive rounds | Claude Sonnet 4.5 · CodeClash | probability of winning next round: 71% overall -> below one third after one loss | Hurt |
| lose a single round | o3 · CodeClash | probability of winning next round: 65% -> 26% | Hurt |
| more edit-behavior (files edited, edit size, turns) | 8 LMs · CodeClash | win rate | No clear effect |
| more frequent inspection of opponent code | GPT-5 vs Claude Sonnet 4.5 · Core War | win rate: 12.8% vs 99.3% access | No clear effect |
| validate edits with arena simulations or tests | Claude Sonnet 4.5 · CodeClash | fraction of rounds validated: 56% | Improved |
Closest papers by meaning
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.89)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.89)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.88)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.88)
- ProgramBench: Can Language Models Rebuild Programs From Scratch? (NeurIPS 2026 · similarity 0.88)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
