Research map / Large language models / Agents, code and math
Nemotron-CORTEXA: Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds NEMOTRON-CORTEXA, an agentic SWE-bench issue-resolution pipeline with a fine-tuned code embedding model (NV-EmbedCode), an AST/LSP-based localization agent ensembled over four LLMs, and diversified patch generation across contexts and edit formats.
Problem
Automatically resolving real-world GitHub issues in large repositories requires localizing the faulty code within limited LLM context and generating correct patches efficiently, which prior pipelines like Agentless handle imperfectly.
Approach
Combine a code-specialized embedding model for file retrieval, a graph-based localization agent with IDE-style navigation tools ensembled over several LLMs, and diversity in patch generation by varying both context and edit format (not only temperature) before test-based filtering and majority voting.
Key results
- Achieves 42.00% (126/300) resolution on SWE-bench Lite and 52.60% (263/500) on Verified at $0.51 per instance, versus Agentless 40.67% and 50.80%.
- NV-EmbedCode reaches 67.67% recall@k on Lite and 71.95% with summaries on Verified, beating NV-Embed-v2 and NV-EmbedQA-v2.
- Localization agent ensemble of four LLMs raises entity recall to 74.72% on Lite versus 59.94% for Agentless.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NEMOTRON-CORTEXA instead of Agentless | SWE-bench software-engineering agent · SWE-bench Lite | resolved: 40.67% -> 42.00% | Helped |
| use NEMOTRON-CORTEXA instead of Agentless | SWE-bench software-engineering agent · SWE-bench Verified | resolved: 50.80% -> 52.60% | Helped |
| use NV-EmbedCode instead of GPT-4o prompt retrieval | file retrieval · SWE-bench Lite | recall@k: 63.00% -> 67.67% | Helped |
| use NV-EmbedCode with an LLM-generated summary instead of the original issue description (ablation) | file retrieval · SWE-bench Lite | recall@k: 67.67% -> 70.33% | Helped |
| use an LLM-generated summary instead of the original issue description (ablation) | BM25 file retrieval · SWE-bench Lite | recall@k: 42.33% -> 38.00% | Hurt |
| use localization agent plus direct prompting instead of Agentless entity localization | entity localization · SWE-bench Lite | recall: 59.94% -> 74.72% | Helped |
| add localization-agent tool use to direct prompting (ablation) | entity localization · SWE-bench Lite | recall: 60.56% -> 71.17% | Helped |
| add direct-prompting entities to localization-agent results (ablation) | entity localization · SWE-bench Lite | recall: 71.17% -> 74.72% | Helped |
| ensemble four localization agents instead of the best individual model (ablation) | entity localization · SWE-bench Lite | recall: 45.94% -> 74.72% | Helped |
| use NV-EmbedCode instead of NV-EmbedQA-v2 to provide files to localization (ablation) | localization agent · SWE-bench Verified | recall: 65.03% -> 67.62% | Helped |
| use entity-localization results instead of the top-1 retrieved file as patch-generation context (ablation) | patch generation · SWE-bench Verified | resolved | Helped |
| vary context and edit format instead of using only LA context and search/replace (ablation) | patch generation · SWE-bench Verified | Pass@9: 270 -> 307 | Helped |
Surprising findings
- Changing edit format and context yields more distinct resolved instances than temperature sampling, which adds fewer new solutions.
- Existing embedding models perform worse with LLM-generated issue summaries than with original descriptions, while NV-EmbedCode improves with summaries.
Limitations
- Majority voting in patch selection is suboptimal when correct solutions are diverse and rarely repeated, as the authors acknowledge.
- Costs for higher-performing submissions are not reported, making cost comparisons with recent systems uncertain.
- Ground truth localization relies on SWE-bench golden patches, which ignores alternative valid fixes elsewhere in the codebase.
Experimental setup
Datasets: SWE-bench training set CoIR (APPS, CoSQA, CodeTransOcean, StackOverflowQA, SyntheticText2SQL) CodeFeedback-derived synthetic code data LLM-generated issue summaries
Benchmarks: SWE-bench Lite SWE-bench Verified
Metrics: % resolved recall@k precision recall pass@1 pass@9 avg cost per instance
Baselines: Agentless OpenHands Moatless Tools AutoCodeRover SWE-agent SFR-Embedding-2 R NV-Embed-v2 NV-EmbedQA-v2 BM25
Builds on: Agentless NV-Embed-v2 information foraging theory SWE-bench
Closest papers by meaning
- OrcaLoca: An LLM Agent Framework for Software Issue Localization (ICML 2025 · similarity 0.88)
- Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.87)
- Structurally Aligned Subtask-Level Memory for Software Engineering Agents (ICML 2026 · similarity 0.87)
- Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks (NeurIPS 2025 · similarity 0.87)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.87)
- Discovering Hierarchical Software Engineering Agents via Bandit Optimization (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.