Research map / Large language models / Agents, code and math
SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Haoyang Huang, Xinping Lei, Yifan Yao, Huaixi Tang, Zhiyi Lai, Kepeng Lei, Zizheng Zhan, Yanan Wu and 26 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| run Claude-Sonnet-4 in Claude Code instead of SWE-Agent (ablation) | Claude-Sonnet-4 · SWE-Compass | average score (%): 31.8 -> 32.9 | Improved |
| run Qwen3-Coder-480B in Claude Code instead of SWE-Agent (ablation) | Qwen3-Coder-480B-Instruct · SWE-Compass | average score (%): 27.2 -> 21.9 | Hurt |
| run Qwen3-Coder-30B in Claude Code instead of SWE-Agent (ablation) | Qwen3-Coder-30B-Instruct · SWE-Compass | average score (%): 20.7 -> 21.6 | Improved |
| choose framework by task: SWE-Agent for bug fixing and feature implementation, Claude Code for test generation and configuration (ablation) | SWE-Agent vs Claude Code · SWE-Compass task types | score | Mixed |
| allow more interaction turns | agentic frameworks · SWE-Compass | Pass@1 | Mixed |
| higher overall performance | evaluated models · SWE-Compass across languages | performance variability (CV) | Improved |
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.91)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.90)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.90)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.90)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.90)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
