Research map / Large language models / Agents, code and math
RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RepoMaster, an agent framework that builds structural graphs and hierarchical trees of GitHub repositories, selects core components, and explores them with pruned context to solve complex tasks, evaluated on MLE-R and the new GitTaskBench.
Problem
LLM code agents struggle to reuse full GitHub repositories for complex real-world tasks because repositories have intricate dependencies and information volume exceeding LLM context limits, leading to high token cost and low task success.
Approach
Combine hybrid structural mapping (hierarchical code tree, function call graph, module dependency graph) with importance-scored core component identification, then let the agent explore the repository with graph-based tools and prune code, documents, and logs to fit a per-turn token budget.
Key results
- On GitTaskBench with Claude 3.5, RepoMaster reaches 62.96% task pass rate versus 40.74% for OpenHands, using about 154k tokens versus 2883k.
- On MLE-R with Claude 3.5, RepoMaster achieves 95.45% valid submissions and 27.27% medal rate (22.73% gold).
- Ablation with GPT-4o shows removing all three mechanisms drops task pass rate from 40.74% to 24.07%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RepoMaster instead of OpenHands | Claude 3.5 agent on GitTaskBench · GitTaskBench | Task Pass Rate (%): 40.74 -> 62.96 | Helped |
| Use RepoMaster instead of OpenHands | Claude 3.5 agent on GitTaskBench · GitTaskBench | #Total Tokens: 2883k -> 154k | Helped |
| Use RepoMaster instead of OpenHands | Claude 3.5 agent on MLE-R · MLE-R | Valid Submission (%): 45.45 -> 95.45 | Helped |
| Add Hybrid Hierarchical Analysis (ablation) | RepoMaster with GPT-4o · GitTaskBench | Task Pass Rate (%): 38.88 -> 40.74 | Helped |
| Add Hybrid Hierarchical Analysis (ablation) | RepoMaster with GPT-4o · GitTaskBench | #Total Tokens: 274k -> 250k | Helped |
| Add Code Exploration (ablation) | RepoMaster with GPT-4o · GitTaskBench | Task Pass Rate (%): 35.18 -> 40.74 | Helped |
| Add Code Exploration (ablation) | RepoMaster with GPT-4o · GitTaskBench | #Total Tokens: 284k -> 250k | Helped |
| Add Information Selection (ablation) | RepoMaster with GPT-4o · GitTaskBench | Task Pass Rate (%): 38.88 -> 40.74 | Helped |
| Add Information Selection (ablation) | RepoMaster with GPT-4o · GitTaskBench | #Total Tokens: 265k -> 250k | Helped |
| Add Hybrid Hierarchical Analysis, Code Exploration, and Information Selection (ablation) | RepoMaster with GPT-4o · GitTaskBench | Task Pass Rate (%): 24.07 -> 40.74 | Helped |
| Add Hybrid Hierarchical Analysis, Code Exploration, and Information Selection (ablation) | RepoMaster with GPT-4o · GitTaskBench | Execution Completion Rate (%): 29.62 -> 48.14 | Helped |
Surprising findings
- The ablated baseline uses fewer tokens than the full system, but this reflects early give-up rather than efficiency.
- Information selection mainly improves signal-to-noise rather than reducing token use, with token usage rising only 6% when removed.
Limitations
- Evaluation on only 22 MLE-R tasks and 54 GitTaskBench tasks across 18 repositories.
- Only Python (.py) files are analyzed; non-source files are removed.
- Relies on closed-source and API models for main results; compute and hardware not reported.
Experimental setup
Datasets: MLE-Bench-Lite (22 tasks, adjusted as MLE-R) GitTaskBench
Benchmarks: MLE-R GitTaskBench
Metrics: valid submission rate medal rate execution completion rate task pass rate total tokens
Baselines: OpenHands SWE-Agent
Builds on: CodeAct AutoGen MLE-Bench
Closest papers by meaning
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.90)
- RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms (NeurIPS 2026 · similarity 0.89)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.89)
- RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · similarity 0.88)
- BootstrapAgent: Turning Repository Setup into Reusable Agent Knowledge (NeurIPS 2026 · similarity 0.88)
- R2E: Turning any Github Repository into a Programming Agent Environment (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.