Research map / Large language models / Instruction and fine-tuning
SelfCodeAlign: Self-Alignment for Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelfCodeAlign, a transparent self-alignment pipeline where a code LLM generates its own coding tasks, multiple test-validated responses, and keeps passing pairs for instruction tuning, yielding a 74k dataset and CodeQwen-7B-based models.
Problem
Instruction tuning for code LLMs typically relies on costly human annotations or distillation from proprietary models, which may be restricted; prior self-instruct attempts for code degraded performance.
Approach
Use the base model alone for the whole pipeline: extract coding concepts from filtered seed functions, generate diverse tasks (Self-OSS-Instruct), sample multiple (response, tests) pairs, execute them in a sandbox, and keep only passing responses for instruction tuning.
Key results
- SelfCodeAlign-CQ-7B reaches 67.1 pass@1 on HumanEval+, 21.4 points above CodeQwen1.5-7B base (45.7) and 16.5 above its OctoPack variant (50.6), and above CodeLlama-70B-Instruct (65.2).
- Execution-based filtering matters: passes-only training gives 65.2 HumanEval+ versus 57.9 for failures-only and 61.6 for random selection.
- Self-generated data outperforms distillation at equal size (74k): 67.1 vs 65.9 for GPT-4o direct distillation, 61.6 for OSS-Instruct, and 59.1 for Evol-Instruct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CodeQwen-7B on SelfCodeAlign self-generated instruction data | CodeQwen-7B · HumanEval+ | pass@1: 45.7 -> 67.1 | Helped |
| train on SelfCodeAlign data instead of OctoPack | CodeQwen-7B · HumanEval+ | pass@1: 50.6 -> 67.1 | Helped |
| use passing responses only instead of random responses (ablation) | SelfCodeAlign · HumanEval+ | pass@1: 61.6 -> 65.2 | Helped |
| use passing responses only instead of failing responses only (ablation) | SelfCodeAlign · HumanEval+ | pass@1: 57.9 -> 65.2 | Helped |
| increase random unfiltered response data from 15.6k to 27.7k examples (ablation) | SelfCodeAlign · HumanEval+ | pass@1: 61.6 -> 61.6 | No clear effect |
| add concept generation between filtered seed functions and instruction generation (ablation) | SelfCodeAlign · HumanEval+ | pass@1: 59.8 -> 65.2 | Helped |
| replace filtered functions with random code snippets for concept generation (ablation) | SelfCodeAlign · HumanEval+ | pass@1: 65.2 -> 64.0 | Hurt |
| use SelfCodeAlign self-generated data rather than GPT-3.5-Turbo Evol-Instruct data | CodeQwen-7B · HumanEval+ | pass@1: 59.1 -> 67.1 | Helped |
| use SelfCodeAlign self-generated data rather than GPT-3.5-Turbo OSS-Instruct data | CodeQwen-7B · HumanEval+ | pass@1: 61.6 -> 67.1 | Helped |
| self-align with the base model's own data rather than GPT-4o responses | CodeQwen-7B · HumanEval+ | pass@1: 65.9 -> 67.1 | Helped |
| generate training data with StarCoder-3 instead of Llama-8 (ablation) | StarCoder-3 · HumanEval+ | pass@1: 34.1 -> 35.4 | Helped |
| generate training data with DeepSeek-33 instead of the base model itself (ablation) | StarCoder-3 · HumanEval+ | pass@1: 35.4 -> 42.1 | Helped |
Surprising findings
- A base model often benefits more from its own generated data than from a stronger teacher's data when the performance gap is small.
- Directly generating instructions from seed snippets without a concept step performed worst (59.8 HumanEval+).
Limitations
- Training data skews toward medium-length samples (~3000-token window), limiting long-context coverage.
- Self-generated unit tests may be erroneous, so some labels of correctness are noisy.
- Negative samples collected during generation are discarded rather than used for preference or RL training.
Experimental setup
Datasets: The Stack V1 (Python functions with docstrings) OctoPack OSS-Instruct dataset Evol-Instruct code dataset
Benchmarks: HumanEval+ MBPP+ LiveCodeBench EvoEval EvalPerf ClassEval DS-1000 CanItEdit
Metrics: pass@1 DPS win-rate
Baselines: CodeQwen1.5-7B-Chat CodeQwen1.5-7B-OctoPack OctoCoder-16B StarCoder2-15B CodeLlama-70B-Instruct OSS-Instruct Evol-Instruct GPT-4o direct distillation DeepSeek-Coder-6.7B-Instruct
Builds on: Self-Instruct OSS-Instruct OctoPack Evol-Instruct MultiPL-T
Closest papers by meaning
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.90)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.89)
- InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.89)
- VERSE: Verification-based Self-Play for Code Instructions (AAAI 2025 · similarity 0.89)
- Toward Verifiable Instruction-Following Alignment for Retrieval Augmented Generation (AAAI 2025 · similarity 0.89)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.