Research map / Large language models / Agents, code and math

Otter: Generating Tests from Issues to Validate SWE Patches

ICML 2025 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Otter, an LLM-based system with a self-reflective action planner and rule-based pre/post-processing that generates fail-to-pass tests from GitHub issues, plus the TDD-Bench-Verified benchmark and a coverage-aware tddScore metric.

Problem

Generating reproduction/acceptance tests from natural-language issue descriptions alone, before the code patch resolving the issue exists, so the tests must fail on old code and pass on the new code.

Approach

A three-stage pipeline (LLM-based test and focal localization, a self-reflective action planner that proposes read/write/modify actions with validation, and a test generator with import fixing), plus an ensemble (Otter++) that mixes heterogeneous prompts and selects tests using execution logs on the old code.

Key results

• Otter with GPT-4o generates fail-to-pass tests for 31.4% of 449 instances; Otter++ (5-test ensemble) reaches 37.0% vs 18.7% for zero-shot baseline.
• On SWT-bench Lite subset (276 instances), Otter++ achieves 29.0% fail-to-pass vs 19.2% for SWE-Agent+.
• Filtering SWE-bench Verified leaderboard patches with Otter++ tests raises precision to 65%-92% at 30%-41% recall for most systems; Otter++ costs about $0.09 per instance with GPT-4o.

Measured effects reported in the paper

Surprising findings

• Heterogeneous prompts with greedy decoding outperformed high-temperature sampling for ensemble diversity (Top-1 166 vs 146).
• Generated tests with high coverage are more likely to be fail-to-pass, while 69 generated tests passed on the old code.

Limitations

• Mined from 12 popular Python repositories, so findings may not generalize to other languages or repos.
• Considers only one test file and generates only one block of code per instance.
• Coverage tool was unreliable for SymPy, so coverage was removed from tddScore for those instances.

Experimental setup

Datasets: SWE-bench SWE-bench Verified SWT-bench Lite

Benchmarks: TDD-Bench-Verified SWT-bench Lite SWE-bench Verified

Metrics: fail-to-pass rate tddScore coverage precision recall cost per sample

Baselines: Zero-shot test file generation LIBRO SWE-Agent SWE-Agent+ AutoCodeRover ZeroShotPlus

Builds on: SWE-bench SWE-bench Verified Agentless CodeT

Closest papers by meaning

• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.87)
• Synthesizing Software Engineering Data in a Test-Driven Manner (ICML 2025 · similarity 0.84)
• TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark (ICLR 2025 · similarity 0.83)
• Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.82)
• SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.82)
• DPIAgent: Divide, Protocol, Isolate for Agentic Reproduction Test Generation (NeurIPS 2026 · similarity 0.82)