Research map / Multi-agent systems and games / LLM agents

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-play SWE-RL (SSR), where a single CWM-based LLM alternates between injecting bugs into real repositories and solving them via RL, using only Docker images with no issues or tests, and outperforms a human-data RL baseline on SWE-bench Verified and SWE-Bench Pro.

Problem

Software agents trained with RL depend on human-curated issue descriptions, tests, and environments, which limits scalability toward superintelligent self-improving agents. The paper asks whether agents can learn software engineering from raw repositories alone via self-play.

Approach

A single LLM policy plays two roles, bug injector and bug solver, in sandboxed repositories. The injector produces formal bug artifacts (bug patch, test-weakening patch, test script and parser), and the solver repairs them. Higher-order bugs built from failed solver attempts and a solve-rate-based injector reward drive an evolving curriculum trained jointly with RL.

Key results

• SSR improves SWE-bench Verified by +10.4 points and SWE-Bench Pro by +7.8 points through self-improvement.
• SSR consistently outperforms the human-data baseline RL across the full training trajectory on both benchmarks.
• Direct bug-injection prompting yields the weakest results, while removal + history-aware injection performs best among injection strategies.

Surprising findings

• Vanilla bug-injection prompting collapses into trivial one-line modifications that provide weak learning signals.
• Solver feedback in the injector reward gives only a slight, largely negligible advantage over a consistency-only reward.

Limitations

• Evaluation noise around 2% paired standard error on SWE-bench Verified.
• Theoretical analysis suggests a sufficiently capable challenger may adopt dominant or tunnel-vision strategies that stall learning.
• Self-play without human language grounding may not improve natural language communication with humans.

Experimental setup

Datasets: SWE-bench Verified SWE-Bench Pro CWM environment Docker images

Benchmarks: SWE-bench Verified SWE-Bench Pro (public split)

Metrics: resolve rate

Baselines: Baseline RL (human-data CWM-style RL) CWM-sft injection-only repair-only direct-injection

Builds on: SWE-RL Code World Model AlphaZero Absolute Zero SPICE

Closest papers by meaning

• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.89)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.88)
• Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.88)
• Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.87)
• daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.87)
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS 2025 · similarity 0.87)