Research map / Multi-agent systems and games / LLM agents

SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-evaluates SWE-Bench Verified agent patches with SWE-ABS, an adversarial two-stage test-strengthening pipeline (coverage-driven augmentation plus mutation-driven adversarial tests), showing weak PR tests inflate leaderboard scores.

Problem

SWE-Bench test suites come from developer PRs and are weak discriminators, so incorrect agent patches pass and inflate resolve rates on SWE-Bench Verified and related benchmarks.

Approach

Two-stage adversarial strengthening: program-slicing-guided coverage augmentation with test decoupling from the gold patch, followed by LLM-generated plausible-but-incorrect mutant patches that evade existing tests, and targeted tests to reject them, with false-negative fixing for over-specific tests.

Key results

• SWE-ABS strengthens 251/500 (50.2%) SWE-Bench Verified instances versus 10/500 for UTBoost (25.1x), rejecting 2,184 of 11,041 previously passing patches (19.78%).
• Top agent TRAE + Doubao-Seed-Code drops from 78.80% to 62.20%, falling from 1st to 5th; Spearman rho falls from 0.98 (UTBoost) to 0.82.
• On SWE-Bench Pro subset, average resolve-rate drop is 16.46 points vs 14.56 on Verified, indicating difficulty and test strength are orthogonal.

Surprising findings

• Task difficulty and test discriminativeness are orthogonal: SWE-Bench Pro, with much lower resolve rates, is strengthened at a comparable rate.
• About one in five passing patches from top-30 agents are semantically incorrect under strengthened tests.

Limitations

• Augmented tests may overfit to the gold patch; 53 instances (10.6%) had overfitting tests needing manual correction.
• Intraprocedural slicing may miss cross-module dependencies.
• Requires a gold patch as reference implementation.

Experimental setup

Datasets: SWE-Bench Verified SWE-Bench Pro

Benchmarks: SWE-Bench Verified SWE-Bench Pro

Metrics: strengthened instances resolve rate drop patch kill count Spearman rank correlation

Baselines: UTBoost

Builds on: UTBoost EvalPlus SWE-Bench

Closest papers by meaning

• The Leaderboard Illusion (NeurIPS 2025 · similarity 0.85)
• AutoAdvExBench: Benchmarking Autonomous Exploitation of Adversarial Example Defenses (ICML 2025 · similarity 0.85)
• Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests (NeurIPS 2026 · similarity 0.84)
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.84)
• Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models (ICML 2026 · similarity 0.84)
• SWE-Marathon: Can AI Agents Autonomously Complete Ultra-Long-Horizon Software Work? (NeurIPS 2026 · similarity 0.84)