Research map / Large language models / Agents, code and math
Pull Requests as a Training Signal for Repo-Level Code Editing
Qinglin Zhu, Tianyu Chen, Shuai Lu, Lei Ji, Runcong Zhao, Murong Ma, Xiangxiang Dai, Yulan He, Lin Gui, Peng CHENG, Yeyun Gong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add repo-level mid-training (StarCoder2-style) before SFT (ablation) | Qwen2.5-Coder-32B · SWE-bench Lite | Pass@1: 11.3 -> 15.7 | Improved |
| Replace StarCoder2-style data with Clean-PR-train (ablation) | Qwen2.5-Coder-32B · SWE-bench Verified | Pass@1: 20.4 -> 30.6 | Improved |
| Use Search/Replace instead of Unified Diff (ablation) | Clean-PR (Python) · SWE-bench Verified | Pass@1: 24.4 -> 27.8 | Improved |
| Add linked issue descriptions to PR context (ablation) | Clean-PR (Python) · SWE-bench Verified | Pass@1: 25.7 -> 27.8 | Improved |
| Train on all 12 languages instead of Python only (ablation) | Clean-PR · SWE-bench Verified | Pass@1: 27.8 -> 30.6 | Improved |
| Add error-driven augmentation to SFT (ablation) | Clean-PR (All languages) · SWE-bench Lite | Pass@1: 21.8 -> 24.3 | Improved |
| Add error-driven augmentation to SFT (ablation) | Clean-PR (All languages) · SWE-bench Verified | Pass@1: 27.4 -> 30.6 | Improved |
| Add error-driven augmentation to SFT (ablation) | Clean-PR (Python) · SWE-bench Lite | File accuracy: 86.7 -> 86.3 | No clear effect |
| Mid-train on StarCoder2-style diffs (ablation) | Qwen2.5-Coder-32B · HumanEval | Pass@1: 54.1% -> 47.6% | Hurt |
| Mid-train on Clean-PR Search/Replace data (ablation) | Qwen2.5-Coder-32B · HumanEval | Pass@1: reaching 59.8% (+5.7%) | Improved |
| Mid-train on Clean-PR Search/Replace data (ablation) | Qwen2.5-Coder-32B · LiveCodeBench | Pass@1: 29.0% -> 32.6% | Improved |
| Sample k=10 patches instead of k=1 | Clean-PR (All languages) · SWE-bench Verified | Pass@k: 30.6 -> 41.5 | Improved |
Closest papers by meaning
- RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · similarity 0.86)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.86)
- RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving (NeurIPS 2025 · similarity 0.86)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.86)
- Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.86)
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
