Research map / Multi-agent systems and games / LLM agents
OrcaLoca: An LLM Agent Framework for Software Issue Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OrcaLoca, an LLM agent framework for software issue localization on SWE-bench Lite that combines priority-based action scheduling, relevance-scored action decomposition, and distance-aware context pruning over a code graph.
Problem
Precisely localizing the code relevant to a GitHub issue in large repositories is hard; existing LLM agents have unstable search planning, rigid graph traversal, and noisy accumulated context.
Approach
Builds a CodeGraph of containment and call edges, schedules LLM-proposed search actions with a priority queue (with counters for repeated actions and redundancy elimination), decomposes class/file search results into method-level sub-actions scored by a sub-agent, and prunes search results by average shortest-path distance to candidate bug locations.
Key results
- OrcaLoca reaches 65.33% function match rate (196/300) on SWE-bench Lite, the best open-source result on the leaderboard snapshot.
- Integrating Agentless-1.5 repair into OrcaLoca yields a 41.00% resolved rate, +6.33 points over Agentless-1.5 reproduction (34.67%).
- Ablation on SWE-bench Common drops function match from 76.34% to 70.97% when disambiguation decomposition is removed, with 3-5 point drops across components.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OrcaLoca localization with Agentless-1.5 patch generation instead of Agentless-1.5 localization | Agentless-1.5 patch-generation pipeline · SWE-bench Lite | Resolved Rate: 34.67% -> 41.00% | Helped |
| Use OrcaLoca localization instead of Agentless-1.5 localization | Agentless-1.5 localization · SWE-bench Lite | Function Match Rate: +6.67 percentage points | Helped |
| Use OrcaLoca localization instead of Agentless-1.5 localization | Agentless-1.5 localization · SWE-bench Lite | Function Match Precision: +4.62 percentage points | Helped |
| Add priority scheduling (ablation) | OrcaLoca · SWE-bench Common | Function Match Rate: 73.12% -> 76.34% | Helped |
| Add file and class decomposition (ablation) | OrcaLoca · SWE-bench Common | Function Match Rate: 72.04% -> 76.34% | Helped |
| Add disambiguation decomposition (ablation) | OrcaLoca · SWE-bench Common | Function Match Rate: 70.97% -> 76.34% | Helped |
| Add distance-aware context pruning (ablation) | OrcaLoca · SWE-bench Common | Function Match Rate: 72.04% -> 76.34% | Helped |
| Batch localization actions with batch size 2 instead of 1 | OrcaLoca · SWE-bench Lite | Token cost per instance: from 0.87 to 0.58 | Helped |
Surprising findings
- Only 38.0% of issues could be reproduced by the auxiliary reproducer agent, limiting its usefulness.
- Batched action execution (batch size 2) reduced localization cost by about 34% without reported loss in localization correctness.
Limitations
- Relies on closed-source Claude 3.5 Sonnet; open-source model generalization is not evaluated.
- Only supports Python repositories; cross-language linking is not handled.
- Ablation uses a smaller 93-instance subset, and gains are within a few instances.
Experimental setup
Datasets: SWE-bench Lite SWE-bench Common SWE-bench Verified
Benchmarks: SWE-bench Lite SWE-bench Common
Metrics: resolved rate function match rate file match rate function match precision
Baselines: Agentless-1.5 AutoCodeRover-v2.0 OpenHands + CodeAct v2.1 SWE-agent RepoGraph HyperAgent Moatless Tools Kodu-v1 Alibaba Lingma Composio SWE-Kit PatchKitty-0.9
Builds on: AutoCodeRover ReAct Chain of Thought Agentless LlamaIndex
Closest papers by meaning
- Nemotron-CORTEXA: Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity (ICML 2025 · similarity 0.88)
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification (ICML 2025 · similarity 0.86)
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution (NeurIPS 2024 · similarity 0.85)
- Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.