Research map / Large language models / Instruction and fine-tuning

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Low-rank adaptation variants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a multi-stage synthetic code-edit data pipeline (127K examples) and SeleKT, a sparse-projection adaptation method, to produce NextCoder code-editing models that retain code generation ability.

Problem

Code LMs struggle with diverse natural-language-driven code edits, and naive fine-tuning on code-edit data causes catastrophic forgetting of code generation and general abilities.

Approach

Periodically select the top-k parameters by magnitude of the task vector (relative to the fixed base model) after dense full fine-tuning, projecting back to a sparse update, instead of fixing the trainable parameters a priori as in PEFT methods.

Key results

• NextCoder-7B (QwenCoder-2.5-7B + SeleKT) reaches 46.1 on CanItEdit and 65.3 on Aider vs 39.3 and 58.7 for the base QwenCoder-2.5-7B, and outperforms DeepSeekCoder-V2-16B on all listed tasks.
• NextCoder-32B scores 74.7% on Aider and 23.6% on Aider Polyglot, versus 18.2% for GPT-4o on Aider Polyglot.
• On HumanEval+/MBPP+, SeleKT keeps QwenCoder-2.5-7B at 84.8/72.0 vs SFT 79.3/67.2 and LoRA 81.7/70.9.

Measured effects reported in the paper

Surprising findings

• Periodically updating the base model during SeleKT (Update θbase variant) performs worse than fixing the base model, contrary to a plausible incremental scheme.
• Standard SFT on code-edit data frequently underperforms the original instruct model on code-editing benchmarks, indicating overfitting.

Limitations

• Dense gradient computation during SeleKT is more costly than parameter-efficient methods.
• Evaluation restricted to code-editing tasks; generality to math or natural language tasks is untested beyond GSM8K/MMLU checks.
• Lower scores on Aider Polyglot show multi-language editing remains difficult.

Experimental setup

Datasets: CommitPackFT StarCoder seed code synthetic code-edit data (GPT-4o and Llama-3.3-70B generated) HumanEvalFix CanItEdit NoFunEval Aider Aider Polyglot HumanEval+ MBPP+ GSM8K MMLU

Benchmarks: CanItEdit HumanEvalFix NoFunEval Aider Aider Polyglot HumanEval+ MBPP+ GSM8K MMLU

Metrics: execution accuracy accuracy NoFunEval score (static analysis, runtime, DiffBLEU)

Baselines: GPT-4o Llama-3-70B-Instruct DeepSeekCoder-33B DeepSeekCoder-V2-16B QwenCoder-2.5-32B LoRA TIES SFT DeepSeek-R1-Qwen-7B Llama-3-8B-Instruct

Builds on: LoRA TIES merging OSS-Instruct CommitPackFT robust fine-tuning (WiSE-FT)

Closest papers by meaning

• Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (ICLR 2025 · similarity 0.91)
• InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.88)
• EditLord: Learning Code Transformation Rules for Code Editing (ICML 2025 · similarity 0.88)
• Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting (AAAI 2025 · similarity 0.88)
• LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.87)
• CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.87)