Research map / Multi-agent systems and games / LLM agents
MaintainCoder: Maintainable Code Generation Under Dynamic Requirements
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MaintainBench, a requirement-change benchmark built from HumanEval, MBPP, APPS, CodeContests, and xCodeEval, and MaintainCoder, a multi-agent waterfall-style pipeline with design-pattern selection that targets maintainable code under evolving requirements.
Problem
Code generators are evaluated on static correctness and ignore maintainability, i.e., how cheaply initial code can adapt to evolving requirements; static metrics like MI and cyclomatic complexity do not capture this dynamic aspect.
Approach
Combine Waterfall-stage agents (requirements analysis, design pattern selection, framework design and evaluation, code generation and optimization) to produce modular code, and measure maintainability dynamically by probing initial code with four requirement-change types and measuring post-change Pass@k, AST similarity, and code diff.
Key results
- MaintainCoder (DeepSeek-V3) reaches 62.5% Pass@5 on APPS-Dyn versus 59.5% for DeepSeek-V3 and 48.0% for MapCoder, with AST similarity 0.828 and relative code diff 29.2%.
- MaintainCoder (GPT-4o-mini) improves xCodeEval original-problem Pass@5 from 46% to 57%.
- On CodeContests-Dyn, MaintainCoder (GPT-4o-mini) lowers cyclomatic complexity to 2.68 versus 6.06 for GPT-4o-mini.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MaintainCoder rather than the GPT-4o-mini baseline | GPT-4o-mini · APPS-Dyn | Pass@5 (%): 35.5 -> 50.5 | Helped |
| Apply MaintainCoder rather than the GPT-4o-mini baseline | GPT-4o-mini · CodeContests-Dyn | Pass@5 (%): 24.2 -> 32.6 | Helped |
| Apply MaintainCoder rather than the GPT-4o-mini baseline | GPT-4o-mini · xCodeEval-Dyn | Pass@5 (%): 23.4 -> 27.3 | Helped |
| Add framework evaluation (ablation) | MaintainCoder with GPT-4o-mini · APPS-Dyn | Pass@5 (%): 49.00 -> 50.50 | Helped |
| Add framework evaluation (ablation) | MaintainCoder with GPT-4o-mini · CodeContests-Dyn | Pass@5 (%): 31.82 -> 32.58 | Helped |
| Add framework evaluation (ablation) | MaintainCoder with GPT-4o-mini · xCodeEval-Dyn | Pass@5 (%): 20.31 -> 27.34 | Helped |
| Add code optimization (ablation) | MaintainCoder with GPT-4o-mini · APPS-Dyn | Pass@5 (%): 40.00 -> 50.50 | Helped |
| Add code optimization (ablation) | MaintainCoder with GPT-4o-mini · CodeContests-Dyn | Pass@5 (%): 28.03 -> 32.58 | Helped |
| Add code optimization (ablation) | MaintainCoder with GPT-4o-mini · xCodeEval-Dyn | Pass@5 (%): 17.19 -> 27.34 | Helped |
| Add framework evaluation (ablation) | MaintainCoder with DeepSeek-V3 · APPS-Dyn | Pass@5 (%): 61.00 -> 62.50 | Helped |
| Add code optimization (ablation) | MaintainCoder with DeepSeek-V3 · APPS-Dyn | Pass@5 (%): 48.50 -> 62.50 | Helped |
Surprising findings
- Static metrics MI and CC contradict each other for multi-agent baselines such as MapCoder, while dynamic metrics are mutually consistent.
- Human programmers produced less maintainable code (AST similarity 0.541, Pass@5 23.5%) than GPT-4o-mini on CodeContests-Dyn.
Limitations
- Benchmark is Python-only and limited to roughly 500 problems across five datasets.
- Phase II modifications rely on a single generator (GPT-4o-mini) for the main results, with only one alternative generator tested.
- Requirement-change simulation is a proxy for real evolution; repository-level tasks such as SWE-Bench are left to future work.
Experimental setup
Datasets: HumanEval MBPP APPS CodeContests xCodeEval
Benchmarks: MaintainBench (HumanEval-Dyn, MBPP-Dyn, APPS-Dyn, CodeContests-Dyn, xCodeEval-Dyn) HumanEval MBPP APPS CodeContests xCodeEval
Metrics: Pass@k Maintainability Index Cyclomatic Complexity AST similarity Code diff (relative and absolute)
Baselines: GPT-4o-mini DeepSeek-V3 Claude-3.5-Sonnet Claude-3.7-Sonnet Gemini-2.5-Flash-Preview GPT-4o Qwen-Plus AgentCoder MapCoder EvoMAC MetaGPT o3-mini
Builds on: AutoGen MapCoder AgentCoder EvoMAC
Closest papers by meaning
- Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation (NeurIPS 2024 · similarity 0.87)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.85)
- Refactoring Code Through Library Design (NeurIPS 2026 · similarity 0.84)
- AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (ICML 2025 · similarity 0.84)
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.84)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.