Research map / Large language models / Instruction and fine-tuning
Training Language Models on Synthetic Edit Sequences Improves Code Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LintSeq, a linter-guided algorithm that refactors code instruction data into synthetic insertion-only edit sequences, and fine-tunes LMs (TinyCodeLM 150M/400M and Gemma 2, Phi-3, Llama 3.1) to synthesize code edit-by-edit.
Problem
Language models synthesize programs in a single autoregressive pass, while human programmers write code by iteratively editing; sequential edit data for code synthesis is scarce, making it hard for LMs to learn edit-by-edit generation and to scale test-time compute efficiently.
Approach
LintSeq backward-samples linter-error-free intermediate program states by random line deletions (with linter-guided removal of affected lines) and converts them via Unix diff into insertion-only edit sequences, so that standard SFT on refactored instruction data teaches models to write code edit-by-edit.
Key results
- TinyCodeLM-150M with LintSeq improves HumanEval pass@1 from 9.1 to 12.8 and MBPP pass@1 from 11.5 to 13.6 over standard SFT.
- Across Gemma 2, Phi-3, and Llama 3.1 models, LintSeq improves HumanEval, MBPP, DS-1000, and BigCodeBench Instruct pass@1 by average absolute gains of +2.3, +4.3, +3.1, and +1.1 respectively.
- Ablating the linter (RandSeq) reduces TinyCodeLM HumanEval pass@1 by roughly 30% (e.g., 6.4 to 4.5 for the 150M model).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune TinyCodeLM-150M on LintSeq edit sequences instead of standard instruction data | TinyCodeLM-150M · HumanEval | pass@1: 9.1 -> 12.8 | Helped |
| Fine-tune TinyCodeLM-400M on LintSeq edit sequences instead of standard instruction data | TinyCodeLM-400M · HumanEval | pass@1: 11.3 -> 13.4 | Helped |
| Fine-tune TinyCodeLM-150M on LintSeq edit sequences instead of standard instruction data | TinyCodeLM-150M · MBPP | pass@1: 11.5 -> 13.6 | Helped |
| Fine-tune TinyCodeLM-400M on LintSeq edit sequences instead of standard instruction data | TinyCodeLM-400M · MBPP | pass@1: 15.5 -> 19.4 | Helped |
| Add linter-guided structure to edit sampling rather than using random edit sampling (ablation) | TinyCodeLM-150M edit-sequence fine-tuning · HumanEval | pass@1: 4.5 -> 6.4 | Helped |
| Add linter-guided structure to edit sampling rather than using random edit sampling (ablation) | TinyCodeLM-400M edit-sequence fine-tuning · HumanEval | pass@1: 6.0 -> 8.4 | Helped |
| Add linter-guided structure to edit sampling rather than using random edit sampling (ablation) | TinyCodeLM-150M edit-sequence fine-tuning · MBPP(+) | pass@1: 6.5 -> 8.6 | Helped |
| Add linter-guided structure to edit sampling rather than using random edit sampling (ablation) | TinyCodeLM-400M edit-sequence fine-tuning · MBPP(+) | pass@1: 10.2 -> 14.2 | Helped |
| Fine-tune on LintSeq edit sequences rather than standard instruction data | Gemma 2, Phi-3, and Llama 3.1 models · HumanEval, MBPP, DS-1000, BigCodeBench Instruct | pass@1: +2.3 HumanEval, +4.3 MBPP, +3.1 DS-1000, and +1.1 BigCodeBench Instruct | Mixed |
| Fine-tune on LintSeq edit sequences rather than standard instruction data | Phi-3, Gemma 2, and Llama 3.1 models · HumanEval, MBPP, CodeContests | pass@k vs total test-time FLOPs | Helped |
Surprising findings
- LintSeq models produce programs with linter errors at a higher frequency than baselines on HumanEval despite higher pass@1.
- Fully random edit sequences (RandSeq) generate buggy code with much higher lint-error frequency than other models.
Limitations
- LintSeq only generates insertion edits, so models cannot perform deletion-based code editing without additional data or RL.
- Experiments are limited to Python code.
- Gains on pass@1 are mixed; HumanEval pass@1 and BigCodeBench greedy pass@1 did not improve for all models.
Experimental setup
Datasets: Magicoder OSS-Instruct Python subset StarCoder2-15B self-alignment instruction data FineWeb 10BT sample The Stack Python subset
Benchmarks: HumanEval MBPP(+) CodeContests DS-1000 BigCodeBench (Instruct)
Metrics: pass@1 pass@k pass@k vs total test-time FLOPs proportion of generations with lint errors
Baselines: CodeT5+ AlphaCode Codex CodeGen-Mono Standard SFT on instruction data RandSeq
Builds on: OSS-Instruct Unix diff OLMo CommitPackFT
Closest papers by meaning
- NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML 2025 · similarity 0.91)
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.89)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.88)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.88)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.88)
- Magicoder: Empowering Code Generation with OSS-Instruct (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.