Research map / Multi-agent systems and games / LLM agents

SERA: Soft-Verified Efficient Repository Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SERA, a low-cost supervised fine-tuning pipeline (SVG) that generates coding-agent trajectories from any repository without test execution, using soft verification by line-level recall, to train open-weight coding agents including a 32B model reaching 49.5%/54.2% on SWE-bench Verified.

Problem

Training open-weight coding agents has required RL infrastructure or complex synthetic bug-injection and test-verification pipelines, making repository specialization and low-budget training impractical.

Approach

Soft Verified Generation: a teacher produces an exploratory fix from a random function with a vague bug prompt, then rewrites it as a synthetic PR and reproduces it in a second rollout; the second patch is checked against the first by line-level recall instead of unit tests, and the resulting trajectories are used for plain SFT.

Key results

• SERA-32B reaches 49.5% ± 1.9% on SWE-bench Verified at 32K context and 54.2% ± 1.4% at 64K, state-of-the-art among fully open-source models per the paper.
• Soft-verified, unverified, and hard-verified trajectories perform similarly at matched scale (Kruskal-Wallis p = .066), suggesting verification is not necessary at these scales.
• Django-specialized SERA-32B reaches 52.23% ± 1.64% on Django instances, matching or exceeding GLM-4.5-Air teacher (51.20%) with 8,000 trajectories.

Surprising findings

• Unverified trajectories (r = 0) perform as well as hard-verified ones, suggesting that test-based verification is unnecessary for SFT data at the scales tested.
• Ordered truncation at high truncation ratios (~0.95) outperforms using fully-fitting trajectories, and random truncation hurts performance.

Limitations

• Evaluation only on SWE-bench Verified; generalization to other coding benchmarks is untested.
• Specialization evaluated on public repositories (Django, Sympy, Sphinx) likely seen in pretraining; private-codebase claims are unverified.
• Statistical power is limited with 3 seeds; several reported differences are within noise.

Experimental setup

Datasets: SWE-smith codebases (121 of 128) SWE-bench Verified demonstration PRs

Benchmarks: SWE-bench Verified

Metrics: resolve rate

Baselines: SWE-smith BugPilot SkyRL-Agent DeepSWE SkyRL-8B Nex-N1-8B Devstral-Small-2 GLM-4.5-Air GLM-4.6 FrogBoss-32B Qwen 3-Coder-30B GLM-4.7-Flash

Builds on: SWE-smith SWE-agent SWE-bench

Closest papers by meaning

• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.88)
• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.87)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.86)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.86)
• Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.85)
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.85)