Research map / Large language models / Agents, code and math
Synthesizing Software Engineering Data in a Test-Driven Manner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-Flow, a test-driven data synthesis framework that builds runtime dependency graphs from unit tests to create incremental development tasks, and the SWE-Flow-Bench benchmark, and fine-tunes Qwen2.5-Coder-32B-Instruct on the synthesized data.
Problem
Existing software engineering data and benchmarks (e.g., SWE-Bench) rely on human-submitted GitHub issues and commit histories, and do not capture incremental, test-driven development of codebases.
Approach
Trace function call relationships during unit test execution to build a Runtime Dependency Graph, derive a topologically ordered development schedule, and skeletonize the codebase at each step so that unit tests verify synthesized TDD tasks without human issues.
Key results
- UF-Coder-32B (Qwen2.5-Coder-32B-Instruct fine-tuned on SWE-Flow data) substantially improves over its base model on SWE-Flow-Bench Lite, e.g., 76% vs 46% replace-format pass rate on arrow.
- Claude-3.5-Sonnet is the strongest agent under OpenHands on SWE-Flow-Bench Lite (e.g., 80% accuracy on arrow), but still fails many tasks.
- Fine-tuning took about two hours on 128 H800 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-Coder-32B-Instruct on SWE-Flow-synthesized data | Qwen2.5-Coder-32B-Instruct · SWE-Flow-Bench Lite, multiple projects and output formats | pass rate: 46.0% -> 76.0%; 30.0% -> 64.0%; 50.0% -> 50.0% | Helped |
Surprising findings
- Claude-3.5-Sonnet performs much better in patch format than replace format, suggesting its training data contains many patch-format examples.
- Most evaluated LLMs perform poorly on patch-format generation, and generated patches often fail to apply with standard patch tools.
Limitations
- The runtime dependency tracing handles only sequential execution and struggles with asynchronous programs and multi-process applications.
- Evaluated on 12 Python projects only, with only 62 projects used for training.
- Even the best agent fails many complex tasks, and evaluation is limited to a single agent framework (OpenHands).
Experimental setup
Datasets: SWE-Flow synthesized training data (16,061 instances)
Benchmarks: SWE-Flow-Bench Lite SWE-Flow-Bench Full
Metrics: pass rate efficiency value
Baselines: SWE-Bench Commit0-Bench Qwen2.5-Coder-32B-Instruct gpt-4o claude-3.5-sonnet DeepSeek-V3 DeepSeek-R1 o1-preview
Builds on: SWE-Bench Megatron-LM OpenHands
Closest papers by meaning
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.87)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.87)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents (NeurIPS 2026 · similarity 0.85)
- Otter: Generating Tests from Issues to Validate SWE Patches (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.