Research map / Large language models / Agents, code and math
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RepoBench, a Python/Java benchmark with three linked tasks (retrieval, next-line completion, end-to-end pipeline) for repository-level code auto-completion, and evaluates retrievers and code LLMs on it.
Problem
Existing code completion benchmarks focus on single-file contexts and do not capture cross-file, multi-file repository-level completion needs, including retrieval of relevant context.
Approach
A benchmark decomposing repository-level completion into three interconnected tasks (RepoBench-R retrieval, RepoBench-C completion with cross-file and in-file context, RepoBench-P end-to-end pipeline), built from import-parsed cross-file dependencies and fresh GitHub repositories to limit leakage.
Key results
- On RepoBench-R, UniXcoder reaches 25.94% acc@1 on Python easy XF-F, outperforming Jaccard (20.82%) and random (15.68%).
- On RepoBench-C-8k Java, Codex (175B) achieves 43.14 EM overall versus 37.35 for StarCoder (15.5B).
- On RepoBench-P with Codex, UniXcoder-L2H retrieval improves Python overall EM to 37.11 from 33.15 for the no-cross-file baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniXcoder rather than random retrieval | RepoBench-R · Python Easy XF-F | acc@1: 15.68 -> 25.94 | Helped |
| Use UniXcoder rather than random retrieval | RepoBench-R · Java Hard XF-F | acc@1: 5.59 -> 10.79 | Helped |
| Fine-tune CodeGen-350M on RepoBench training data (ablation) | CodeGen-350M · RepoBench-C Python All | EM: 20.71 -> 20.85 | Helped |
| Fine-tune CodeGen-2.7B on RepoBench training data (ablation) | CodeGen-2.7B · RepoBench-C Python All | EM: 27.35 -> 26.66 | Hurt |
| Add cross-file context to the in-file-only pipeline | Codex RepoBench-P · Python All | EM: 33.15 -> 34.15 | Helped |
| Use UniXcoder-L2H retrieval rather than random retrieval | Codex RepoBench-P · Java All | EM: 40.77 -> 41.83 | Helped |
| Add import statements to short in-file context (ablation) | Codex prompt construction · Python All | EM: 26.64 -> 33.21 | Helped |
| Add cross-file context and import statements to short in-file context (ablation) | Codex prompt construction · Java All | EM: 32.69 -> 46.62 | Helped |
| Use long rather than short in-file context (ablation) | Codex prompt construction · Python XF-R | EM: 36.28 -> 43.44 | Helped |
| Increase retained retrieval context from 3 to 120 lines (ablation) | UniXcoder retrieval · Python Easy XF-F | acc@1: 27.02 -> 16.09 | Hurt |
| Increase retained retrieval context from 3 to 120 lines (ablation) | UniXcoder retrieval · Java Easy XF-F | acc@1: 20.44 -> 12.31 | Hurt |
| Increase retained retrieval context from 3 to 120 lines (ablation) | Jaccard retrieval · Python Easy XF-F | acc@1: 18.10 -> 15.02 | Hurt |
Surprising findings
- StarCoder often performs best on RepoBench-P with no retrieval (Baseline), suggesting limited repository-level generalization.
- Jaccard similarity generally beats edit similarity for lexical retrieval.
Limitations
- Many models run quantized with CTranslate2, which may cause discrepancies from original model results.
- Codex results are from a deprecated model and rate-limited API, so random retrieval is single-run and results are indicative only.
- RepoBench-P uses down-sampling and minimum token thresholds, and Codex results are not averaged over multiple runs.
Experimental setup
Datasets: github-code newly crawled GitHub repositories (Feb-Aug 2023)
Benchmarks: RepoBench-R RepoBench-C RepoBench-P
Metrics: acc@k exact match edit similarity
Baselines: Codex CodeGen StarCoder CodeBERT UniXcoder Jaccard similarity Edit similarity
Builds on: RepoCoder CoCoMIC CodeXGLUE
Closest papers by meaning
- Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.89)
- Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs (AAAI 2025 · similarity 0.88)
- RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms (NeurIPS 2026 · similarity 0.87)
- RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving (NeurIPS 2025 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.85)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.