Research map / Multi-agent systems and games / LLM agents

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SlopCodeBench, a 36-problem, 196-checkpoint iterative benchmark where coding agents extend their own code under evolving specs, measuring structural erosion and verbosity, and finds agents degrade code quality faster than human repositories.

Problem

Single-shot coding benchmarks hide how agent design decisions degrade code quality over iterative extensions; existing iterative benchmarks constrain the design space or reuse human workspaces.

Approach

Agents extend their own prior workspace across checkpoints with black-box specs (CLI/API only) and hidden tests, while trajectory-level metrics (erosion via complexity mass in high-CC functions, verbosity via ast-grep flags and clone lines) track code degradation at each step, calibrated against 473 human Python repositories.

Key results

• Best agent (GPT 5.5) passes 14.8% of checkpoints strictly; no agent fully solves any problem end-to-end.
• Structural erosion rises in 77% of agent trajectories and verbosity in 75.5%.
• Agent code is 2.3x more verbose and 2.0x more eroded than the 473-repository human panel; agent verbosity grows about 6.6x and erosion about 5.0x faster per checkpoint.
• Anti-slop prompt cuts erosion and verbosity substantially (e.g., GPT 5.5 erosion 0.49 to 0.21) but lowers strict solve rate (14.8% to 9.2%).

Surprising findings

• Quality-aware prompts reduce initial verbosity and erosion but do not slow iterative degradation.
• Pass rates stay relatively stable across checkpoints for functionality tests while error and core pass rates drop sharply.

Limitations

• Only the Python track is evaluated due to cost.
• Metrics are static proxies (cyclomatic complexity mass, ast-grep rules) and may not capture all quality aspects.
• Evaluation uses native harnesses with varying versions, limiting controlled comparison.

Experimental setup

Datasets: 473 open-source Python repositories (git histories)

Benchmarks: SlopCodeBench

Metrics: strict solve rate isolated solve rate core solve rate partial solve rate structural erosion verbosity cost per checkpoint

Baselines: Just-Solve prompt Anti-Slop prompt Plan-First prompt SWE-agent

Builds on: SWE-agent Terminal Bench Radon cyclomatic complexity ast-grep

Closest papers by meaning

• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.93)
• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.92)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.91)
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.90)
• Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.90)
• SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.89)