Research map / Large language models / Agents, code and math
Software engineering agents
Build and evaluate coding agents using execution feedback and code-graph models.
140 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Agents, code and math, keyword cluster “evaluating · capabilities · bench”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 119 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 5.8% to 9.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback (NeurIPS 2023 · 14 citations)
- 2024 · RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR 2024 · 12 citations)
- 2025 · AnalogCoder: Analog Circuit Design via Training-Free Code Generation (AAAI 2025 · 55 citations)
- 2026 · MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers (ICLR 2026 · 4 citations)
Earliest papers here
- InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback (NeurIPS 2023 · 14 citations)
- CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion (NeurIPS 2023 · 11 citations)
- SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models (NeurIPS 2023 · 6 citations)
What has been tried
55 papers here report 557 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 4%, hurt 29%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use Python code generation instead of SPICE code generation | Analog circuit benchmark · Pass@1 | 13.9 -> 21.4 | Helped | AnalogCoder: Analog Circuit Design via Training-Free Code Generation (AAAI 2025 · 55 citations) |
| Add domain-adaptive pretraining | Test dataset, VEXCategory · micro-F1 | 0.83 -> 0.94 | Helped | CVE-LLM: Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models (AAAI 2025 · 20 citations) |
| Aggregate feedback in 1 batch instead of 0 | BIRD dev · EX | 56.52 -> 57.4 | Helped | MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL (AAAI 2025 · 17 citations) |
| Fine-tune CodeGen-350M on RepoBench training data | RepoBench-C Python All · EM | 20.71 -> 20.85 | Helped | RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (ICLR 2024 · 12 citations) |
| add accurate image captions | EEE-Bench · accuracy | 46.78 -> 45.35 | Hurt | EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark (CVPR 2025 · 11 citations) |
| Add agent-proposed attacks to the human-initialized attack library | HarmBench, Llama-3.1-70B · ASR | 0.75 -> 0.82 | Helped | AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration (NeurIPS 2025 · 11 citations) |
| Add data filtered by the semantic checker to the training set | BFCL · overall accuracy | Hurt | APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets (NeurIPS 2024 · 8 citations) | |
| Use few-shot prompting | CodeContests valid, 1@3 · solve rate | 8.9 -> 8.5 | Hurt | RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · 6 citations) |
| increase reasoning effort from low to high | IC SWE (Diamond) · pass@1 | 9.3% -> 16.5% | Helped | SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (ICML 2025 · 4 citations) |
| use multi-round inference | SpreadsheetBench · soft restriction overall | 0.51 -> 3.66 | Helped | SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation (NeurIPS 2024 · 4 citations) |
| remove program abstraction (RepoAudit-NoAbs) | NPD/MLK/UAF 15 projects · true positives and false positives | TPs 13+8 vs 21+19 ... precision drops to 40.38%; FPs rise to 31 (vs full) | Hurt | RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · 3 citations) |
| Increase repair samples from 20 to 160 | SWE-bench Verified · pass@1 | 33.6 to 40.0 | Helped | SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 63 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HumanEval · 5 of 63 papers
- MBPP · 3 of 63 papers
- SWE-bench Verified · 3 of 63 papers
- SWE-bench training set · 3 of 63 papers
- SWE-bench · 2 of 63 papers
- CrossCodeEval · 2 of 63 papers
Baselines
- SWE-agent · 9 of 63 papers
- GPT-4o · 8 of 63 papers
- Agentless · 7 of 63 papers
- OpenHands · 7 of 63 papers
- Claude-3.5-Sonnet · 6 of 63 papers
- DeepSeek-R1 · 6 of 63 papers
- DeepSeek-V3 · 5 of 63 papers
- AutoCodeRover · 3 of 63 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 140 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 10 papers
- Tsinghua University · 7 papers
- University of California, Berkeley · 6 papers
- Stanford University · 5 papers
- Alibaba Group · 5 papers
- Meta · 5 papers
- University of Illinois, Urbana Champaign · 5 papers
- Amazon · 4 papers
- The University of Hong Kong · 4 papers
- Chinese Academy of Sciences · 4 papers
Use it
Download BibTeX of 23 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Agent instruction-following benchmarks (176 papers)
- Long-horizon LLM agents (155 papers)
- Reasoning benchmarks and platforms (47 papers)
- LLM-driven embodied and driving agents (54 papers)
Elsewhere in the topic
- Mathematical and structural reasoning benchmarks (304 papers)
- Inductive and abstract reasoning evaluation (158 papers)
- LLMs for optimization and heuristics (147 papers)
- LLMs for scientific discovery (100 papers)
- Code generation benchmarks (96 papers)
- LLM-guided program synthesis (93 papers)
- Execution-based code verification (90 papers)
- Retrieval-augmented formal theorem proving (62 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).