Research map / Multi-agent systems and games / LLM agents
RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RefactorBench, 100 handcrafted multi-file Python refactoring tasks with AST-based subtask tests, shows LM agents struggle (SWE-agent ~22% base), and improves them with state-aware interfaces that condition on a natural-language summary of prior edits.
Problem
LM agents are mainly evaluated on GitHub issue-solving, which is noisy and skews task styles; their stateful reasoning and action tracking across multi-file code refactors is under-measured.
Approach
A benchmark of multi-file refactoring tasks with three instruction specificity levels (lazy, base, descriptive), AST-based subtask tests, and a state-aware interface where an externally computed natural-language state summary of prior edits is appended before each agent action.
Key results
- SWE-agent with gpt-4 solves 12%, 18%, and 27% of lazy, base, and descriptive RefactorBench tasks; a human developer solves 87% with base instructions.
- State-aware SWE-agent yields an average 43.9% relative improvement in task success over baseline agents across instruction sets.
- Subtask completion rises by an average of 71% with state-aware interfaces.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add recurring representations of previous edits to the SWE-agent interface | SWE-agent with GPT-4o · RefactorBench | task success rate: 43.9\% relative increase | Helped |
| Add recurring representations of previous edits to the SWE-agent interface | SWE-agent with GPT-4o · RefactorBench | subtask completion rate: 71\% increase | Helped |
Surprising findings
- No tasks requiring changes in more than 6 files are solved by any baseline.
- About 44% of tests checking for changes show agents did not edit target files even when the descriptive instruction named them.
Limitations
- Only Python repositories; tasks limited to 100 instances due to construction cost.
- Gold reference solutions withheld; results are non-deterministic across runs.
- Baselines use modified SWE-agent prompts that remove bug-reproduction and testing, treated as upper bounds.
Experimental setup
Datasets: RefactorBench
Benchmarks: RefactorBench
Metrics: task success rate (all AST tests pass) subtask completion rate
Baselines: SWE-agent SWE-agent with prompt adjustment
Builds on: SWE-agent SWE-bench
Closest papers by meaning
- Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.89)
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.89)
- PACE: Two-Timescale Self-Evolution for Small Language Model Agents (NeurIPS 2026 · similarity 0.89)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.89)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.88)
- LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.