Research map / Multi-agent systems and games / LLM agents

RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RepoMirage, a perturbation-based two-stage evaluation on SWE-Bench Verified showing code agents drop sharply on repository context reasoning tasks, and proposes RepoAnchor, a structure-first two-stage workflow that partially mitigates the gap.

Problem

Success on repository-level issue-resolution benchmarks may not reflect genuine repository context reasoning (identifying and relating task-relevant information across files), and existing end-to-end evaluations do not isolate this capability.

Approach

Use semantics-preserving repository perturbations (dependency-path indirection, runtime-target masking, local-value externalization) to raise the demand for cross-file context, then convert the resulting bottlenecks into explicit verifiable tasks; diagnose "exploration drift" and propose a structure-first explore-then-solve workflow (RepoAnchor).

Key results

• Across eight models, average resolved rate drops from 66.80% on original SWE-Bench Verified to 49.78% under RepoMirage-Perturb, with average accessed files rising from 4.77 to 13.24.
• On RepoMirage-Extend, average success falls from 66.80% to 25.25% across models.
• Structural hints raise GPT-5 average on RepoMirage-Extend from 27.60% to 62.60% and DeepSeek-V3.2 from 39.80% to 76.80%.

Surprising findings

• Agents explore more files under perturbation but transition from exploration to editing less often, a pattern the authors call exploration drift.
• Some models that perform strongly on original issue resolution (e.g., MiniMax-M2.7) drop sharply on the perturbation-derived tasks, so benchmark scores do not align with the new capability.

Limitations

• Construction limited to Python repositories.
• Tasks follow a classical problem-solving format; other SE scenarios (code review, test generation, refactoring) not covered.
• RepoAnchor is described as an initial prototype; gains are demonstrated but mitigation remains partial.

Experimental setup

Datasets: SWE-bench Verified

Benchmarks: SWE-bench Verified RepoMirage-Perturb RepoMirage-Extend

Metrics: resolved rate average number of accessed files task success rate transition probability

Baselines: GPT-4.1 GPT-5 Gemini-3.1-Pro Claude-Sonnet-4.6 DeepSeek-V3.2 MiniMax-M2.7 Qwen3-Coder-Next Qwen3.6-35B-A3B

Builds on: SWE-bench Verified mini-swe-agent LibCST

Closest papers by meaning

• Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.90)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.89)
• BootstrapAgent: Turning Repository Setup into Reusable Agent Knowledge (NeurIPS 2026 · similarity 0.88)
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.87)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.87)
• RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · similarity 0.87)