Research map / Multi-agent systems and games / LLM agents

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SWE Atlas, a 284-task expert-authored benchmark for coding agents on Codebase Q&A, Test Writing, and Refactoring, scored with programmatic checks plus rubric-based LLM-as-judge grading.

Problem

Existing SWE benchmarks focus on issue resolution and feature implementation, leaving underrepresented professional workflows (codebase comprehension, test authoring, refactoring) and engineering quality (maintainability, cleanliness) under-evaluated.

Approach

Three under-specified, agentic task categories built from 18 active open-source repos, each with category-specific hybrid evaluation: mutation checks and manifest checks for tests, behavior-preservation regression tests for refactors, and expert-written must-have rubrics graded by an LLM judge.

Key results

• GPT 5.4 (Codex) achieves the highest Pass@1 at 43.49% and Opus 4.7 (Claude Code) is close at 41.89%; the best open-weight model GLM 5 reaches 24.03% under mini-SWE-agent.
• Pass^3 drops 30-50% from Pass@1 for the best configurations, indicating inconsistency across trials.
• Refactoring tasks show a 15-40 point gap between regression-test pass rate and must-have rubric pass rate, so test-based grading overstates engineering quality.

Surprising findings

• Test-based functional checks overstate refactoring quality: models pass regression tests 60-80% of the time but score much lower on rubric checks.
• Gemini models almost never spawn sub-agents on Codebase Q&A (about 1% of trials), while Claude models spawn them in nearly every trial.

Limitations

• Single-turn tasks only; multi-turn clarification and steering are not evaluated.
• Reliance on LLM-as-a-judge rubric grading introduces grading bias despite agreement checks.
• Copyleft open-source repositories carry residual contamination risk; internet access was enabled during evaluation.

Experimental setup

Datasets: SWE Atlas

Benchmarks: SWE Atlas

Metrics: Pass@1 Pass^3 Pass@3 rubric pass rate mutation kill rate Cohen's kappa Macro-F1 Jaccard similarity

Baselines: GPT 5.3 Codex Opus 4.6 Sonnet 4.6 Gemini 3.1 Pro Gemini 3 Flash GLM 5 Kimi K2.5 Minimax M2.5

Builds on: SWE-Bench Pro Harbor framework mini-SWE-agent

Closest papers by meaning

• Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.92)
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.90)
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.90)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.89)
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.89)
• SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades (NeurIPS 2026 · similarity 0.89)