Research map / Multi-agent systems and games / LLM agents
R2E: Turning any Github Repository into a Programming Agent Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents R2E, a framework that turns any GitHub Python repository into an executable programming-agent environment by synthesizing equivalence test harnesses, and instantiates it as the R2E-Eval1 benchmark (246 problems).
Problem
Existing code generation benchmarks consist of short isolated functions without repository context or executable functional tests, so they do not evaluate realistic, interactive programming with real-world codebases.
Approach
Use the ground-truth repository function as a reference implementation to generate equivalence tests (decoupling expected outputs from LLM prediction), build equivalence test harnesses with setup for dependencies, and use dependency slicing to provide minimal repository context for test generation and problem prompts.
Key results
- GPT-4 achieves pass@1 close to 50% on R2E-Eval1, while other models reach roughly 30%, and performance is lower than on HumanEval.
- Equivalence test generation with dependency-sliced context yields about 44% valid harnesses with about 83% branch coverage (sliced strategy 52.37% in-file validity).
- Self-repair with interpreter and test-harness feedback raises GPT-4 pass rate on initially failing problems to a maximum of 33% after 5 iterations (GPT-3.5-Turbo 20%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use equivalence tests instead of predicting expected outputs | R2E test generation · In-File | validity: 35.43% -> 44.25% | Helped |
| use equivalence tests instead of predicting expected outputs | R2E test generation · Out-File | validity: 30.68% -> 19.82% | Helped |
| add dependency-sliced context (ablation) | R2E equivalence test generation · In-File | validity: 51.73% -> 52.37% | Helped |
| add dependency-sliced context (ablation) | R2E equivalence test generation · Out-File | validity: 32.24% -> 35.01% | Helped |
| add dependency-sliced context (ablation) | R2E equivalence test generation · Out-File | branch coverage: 77.23% -> 79.65% | Helped |
| add dependency-sliced context (ablation) | R2E equivalence test generation · In-File | branch coverage: 87.32% -> 88.18% | Helped |
| add full repository context rather than no context (ablation) | R2E equivalence test generation · Out-File | validity: 19.82% -> 32.24% | Helped |
| add interpreter and test-harness error feedback over 5 iterations | GPT-4 self-repair · R2E-Eval1 failing instances | self-repair rate: 0% -> 33% | Helped |
| add interpreter and test-harness error feedback over 5 iterations | GPT-3.5-TURBO self-repair · R2E-Eval1 failing instances | self-repair rate: 0% -> 20% | Helped |
| add zero-shot or two-shot chain-of-thought prompting | GPT-3.5-TURBO and GPT-4 · R2E-Eval1 | PASS@1 | No clear effect |
Surprising findings
- Chain-of-thought prompting does not improve performance over direct prompting for GPT-3.5-Turbo and GPT-4 on R2E-Eval1.
- Models often re-implement functionality already present in context instead of reusing existing functions.
Limitations
- Natural-language docstrings can be ambiguous and under-specify corner cases.
- Branch coverage is a soft check for test quality; equivalence is checked only over a finite set of inputs.
- Test generation validity is only about 44%, requiring filtering and manual inspection.
Experimental setup
Datasets: GitHub Python repositories (429 installed, created after July 2022)
Benchmarks: R2E-Eval1
Metrics: pass@1 validity branch coverage
Baselines: HumanEval ODEX CrossCodeEval RepoBench RepoEval SWE-bench GPT-3.5-Turbo CodeLlama
Builds on: HumanEval+ self-instruct chain-of-thought Reflexion
Closest papers by meaning
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.90)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.89)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.88)
- RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving (NeurIPS 2025 · similarity 0.87)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.87)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.