Research map / Large language models / Instruction and fine-tuning
LLM-Assisted Code Cleaning For Training Accurate Code Generators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses GPT-3.5-turbo to clean existing algorithmic code datasets (renaming, modularization, plan annotation) verified by test cases, then fine-tunes CodeLlama-7B on the cleaned data, showing up to 30% gains on APPS and Code-Contests.
Problem
Code generation models are trained on datasets judged only by functional correctness, ignoring readability and structure; the paper asks whether improving code style and structure improves training data quality for algorithmic code generation.
Approach
Instead of generating new synthetic data, transform existing programs into parallel cleaned versions via LLM instructions (rename variables, modularize into helper functions, prepend natural-language plans), using test-case equivalence checks to keep functional correctness.
Key results
- Fine-tuning CodeLlama-7B on D_modular raises APPS-Introductory pass@1 from 18.7 to 22.7 (about 23% relative), and Code-Contests pass@25 from 6.4 to 8.3-8.4.
- Training on 15% of the cleaned modular data matches fine-tuning on the full original APPS dataset in pass@1.
- GPT-4 judge rates 99.4% of 998 transformed programs as consistent with originals; variable names average 2.96/3 and decomposition 2.93/3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original training programs with modularized programs | CL-7B fine-tuning · APPS-Introductory | pass@1: 18.7 -> 22.7 | Helped |
| Replace original training programs with modularized programs | CL-7B fine-tuning · Code-Contests | pass@25: 6.4 -> 8.3 | Helped |
| Use modularized examples instead of original examples for in-context learning | CL-7B · APPS-Introductory | pass@1: 14.2 -> 17.5 | Helped |
| Use modularized examples instead of original examples for in-context learning | CL-7B · Code-Contests | pass@100: 7.2 -> 9.3 | Helped |
| Add generated plan annotations to modularized training programs (ablation) | CL-7B fine-tuning · APPS-Introductory | pass@25: 42.6 -> 43.8 | Helped |
| Add generated plan annotations to modularized training programs (ablation) | CL-7B fine-tuning · Code-Contests | pass@100: 12.4 -> 10.8 | Hurt |
| Add ground-truth plans at inference | CL-7B trained on planning data · Code-Contests-Plan | pass@100: 17.8 -> 28.1 | Helped |
| Rename variables descriptively without modularizing (ablation) | CL-7B fine-tuning · APPS-Introductory | pass@1: 17.2 -> 19.1 | Helped |
| Rename variables descriptively without modularizing (ablation) | CL-7B fine-tuning · Code-Contests | pass@10: 5 -> 4.7 | Hurt |
| Train on GPT-3.5-turbo-generated solutions instead of cleaned existing solutions (ablation) | CL-7B fine-tuning · APPS-Introductory | pass@1: 22.7 -> 21.1 | Hurt |
| Use GPT-4-Turbo instead of GPT-3.5-turbo for transformations (ablation) | CL-7B trained on modularized data · APPS-Introductory | pass@10: 33.0 -> 34.3 | Helped |
| Train on 15% of the modularized dataset instead of the full original dataset (ablation) | CL-7B fine-tuning · APPS | pass@1 | No clear effect |
Surprising findings
- Planning annotations give only marginal or no gains over modularization, and renaming alone gives small gains.
- Fine-tuning on distilled GPT-3.5-turbo solutions (D_distill) underperforms fine-tuning on cleaned modular data, despite the distilled data coming from a stronger model.
Limitations
- Transformations rely on GPT-3.5-turbo with no correctness guarantees beyond test-case equivalence; some helper names are improper or implementations are copied wholesale.
- Planning gains are limited; generated natural-language plans are often imprecise or incorrect.
- Evaluation limited to algorithmic code generation with one 7B model; fine-tuning gains over in-context learning are modest.
Experimental setup
Datasets: APPS Code-Contests
Benchmarks: APPS-Introductory APPS-Interview APPS-Competition Code-Contests
Metrics: pass@1 pass@10 pass@25 pass@100
Baselines: CodeLlama-7B fine-tuned on original data GPT-3.5-turbo distillation (D_distill) AlphaCode-9B AlphaCode-41B Code-Davinci-002 GPT-3.5-turbo
Builds on: CodeLlama chain-of-thought prompting Parsel AlphaCode dataset
Closest papers by meaning
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
- Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.