Research map / Large language models / Instruction and fine-tuning
InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Inverse-Instruct, a self-augmentation method where a fine-tuned code LLM summarizes its own training code snippets into new instructions, self-selects the best one via YES pseudo-probability, and retrains to produce InverseCoder code models.
Problem
Instruction-tuned open code LLMs depend on data generated by expensive closed-source LLMs; the paper asks whether a fine-tuned open model can improve itself by generating additional instruction data.
Approach
A code snippet can serve as the response to many instructions, and code-to-instruction summarization is easier for code LLMs than instruction-to-code generation. The model summarizes clean code from its own training set into instructions, self-evaluates them via YES pseudo-probability, and fine-tunes on the augmented data.
Key results
- InverseCoder-DS-6.7B reaches 76.8% HumanEval+, 69.0% MBPP+, 62.6% MultiPL-E and 44.2% DS-1000, reported as SOTA among fully open (model and data) models of 6.7B size.
- InverseCoder-CL-7B improves HumanEval+ from 69.5 (WizardCoder-GPT4-CL-7B, 72.6/68.9 reported) to 76.2 (72.0) using self-generated data.
- Ablation: removing the self-evaluation/selection step drops HumanEval(+) from 76.2 (72.0) to 75.6 (71.3) for CL-7B; top-3 instruction selection drops to 70.1 (67.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Inverse-Instruct data augmentation | CodeLlama-Python-7B · HumanEval+ | pass@1: 68.9 -> 72.0 | Helped |
| Apply Inverse-Instruct data augmentation | DeepSeek-Coder-6.7B · MBPP+ | pass@1: 67.5 -> 69.0 | Helped |
| Add self-evaluation and data selection after code summarization (ablation) | Inverse-Instruct on CodeLlama-Python-7B · HumanEval+ | pass@1: 71.3 -> 72.0 | Helped |
| Add self-evaluation and data selection after code summarization (ablation) | Inverse-Instruct on CodeLlama-Python-7B · MBPP+ | pass@1: 58.2 -> 60.1 | Helped |
| Add preprocessing to the full code-summarization and evaluation pipeline (ablation) | Inverse-Instruct on CodeLlama-Python-7B · HumanEval+ | pass@1: 67.1 -> 72.0 | Helped |
| Generate instructions by summarizing code instead of generating responses from instructions (ablation) | Inverse-Instruct on CodeLlama-Python-7B · HumanEval+ | pass@1: 67.1 -> 72.0 | Helped |
| Use YES pseudo-probability for data selection instead of random selection (ablation) | Inverse-Instruct on CodeLlama-Python-7B · HumanEval+ | pass@1: 68.3 -> 72.0 | Helped |
| Select the top 3 instructions per response instead of top 1 (ablation) | Inverse-Instruct on CodeLlama-Python-7B · HumanEval+ | pass@1: 72.0 -> 67.1 | Hurt |
| Apply Inverse-Instruct a second round using InverseCoder-generated data (ablation) | InverseCoder-CL-7B · HumanEval+ | pass@1: 72.0 -> 70.1 | Hurt |
| Add self-generated code after generating instructions from code (ablation) | Inverse-Instruct on CodeLlama-Python-7B · HumanEval+ | pass@1: 72.0 -> 68.9 | Hurt |
| Train on self-generated instructions before GPT-4 data rather than the reverse order (ablation) | InverseCoder-CL-7B · HumanEval+ | pass@1: 59.1 -> 72.0 | Helped |
Surprising findings
- Selecting multiple self-generated instructions per response (top-3, top-5) hurts performance, so only the single best instruction is kept.
- Applying Inverse-Instruct a second time to InverseCoder gives no significant gain, consistent with model collapse from repeated self-generated training.
Limitations
- Method may perform worse on models with weak Code-to-NL ability.
- Requires access to a large high-quality code snippet corpus.
- Repeated self-augmentation yields no significant gain (model collapse risk).
Experimental setup
Datasets: evol-codealpaca-v1 Magicoder-OSS-Instruct-75K MBPP+
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ MultiPL-E DS-1000
Metrics: pass@1
Baselines: WizardCoder-GPT4 Magicoder WaveCoder-Ultra AlchemistCoder StarCoder2-Instruct GPT-4-Turbo GPT-3.5-Turbo DolphCoder DIT
Builds on: Self-Instruct Evol-Instruct OSS-Instruct Magicoder AutoMathText
Closest papers by meaning
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.89)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.89)
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.89)
- InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining (ICML 2024 · similarity 0.88)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
- NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.