Research map / Large language models / Agents, code and math
CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
Alex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a plan-then-implement stage (ablation) | GPT-5.2 · CodeTaste Open Track | alignment score (%): 7.9 -> 15.1 | Improved |
| add a plan-then-implement stage (ablation) | Claude Sonnet 4.5 · CodeTaste Open Track | alignment score (%): 6.2 -> 10.7 | Improved |
| add a plan-then-implement stage (ablation) | GPT-5.1 Codex Mini · CodeTaste Open Track | alignment score (%): 7.2 -> 9.8 | Improved |
| add a plan-then-implement stage (ablation) | MiniMax M2.7 · CodeTaste Open Track | alignment score (%): 7.0 -> 3.4 | Hurt |
| add a plan-then-implement stage (ablation) | Qwen3 · CodeTaste Open Track | alignment score (%): 2.0 -> 2.1 | No clear effect |
| add oracle selection among multiple plans (ablation) | GPT-5.2 · CodeTaste Open Track | alignment score (%): 15.1 -> 20.6 | Improved |
| add oracle selection among multiple plans (ablation) | Claude Sonnet 4.5 · CodeTaste Open Track | alignment score (%): 10.7 -> 10.1 | No clear effect |
| add oracle selection among multiple plans (ablation) | GPT-5.1 Codex Mini · CodeTaste Open Track | alignment score (%): 9.8 -> 4.2 | Hurt |
| add plan stage (test pass rate) (ablation) | Qwen3 · CodeTaste Open Track | Pass (%): 58.0 -> 35.0 | Hurt |
| add plan stage (test pass rate) (ablation) | GPT-5.2 · CodeTaste Open Track | Pass (%): 87.0 -> 87.0 | No clear effect |
| use randomized six-plan judge selection instead of oracle multiplan (ablation) | GPT-5.2 · CodeTaste Open Track | IFR (%): 25.5 -> 26.1 | No clear effect |
| give detailed refactoring instructions instead of focus area only | GPT-5.2 · CodeTaste | alignment score (%): 7.9 -> 69.6 | Improved |
Closest papers by meaning
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.88)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.87)
- Code Repair with LLMs gives an Exploration-Exploitation Tradeoff (NeurIPS 2024 · similarity 0.87)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.87)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.87)
- Copilot Arena: A Platform for Code LLM Evaluation in the Wild (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
