Research map / Large language models / Instruction and fine-tuning
AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Code LLMs (Llama 2-7B, CodeLlama-Python-7B, DeepSeek-Coder-Base-6.7B) on a harmonized multi-source code dataset using GPT-4-generated data-specific AlchemistPrompts and code comprehension tasks.
Problem
Single-source fine-tuning data for Code LLMs has limited quality and diversity, while naively mixing multi-source code data causes conflicts in languages and response styles that can hurt performance.
Approach
Generate data-specific "AlchemistPrompts" with GPT-4 that describe response characteristics and prepend them to instructions (hindsight relabeling) to harmonize conflicting multi-source data, and add code comprehension tasks (instruction evolution, data filtering, code review) built from the data construction process as training data.
Key results
- AlchemistCoder-DS (6.7B) reaches 79.9% pass@1 on HumanEval and 77.0% on MBPP, the best among 6.7B/7B models in the paper's table.
- AlchemistCoder-CL achieves 40.3% on DS-1000 and 57.3% average on HumanEval-X, versus 35.4% for CodeLlama-Python-7B on HumanEval-X.
- Overall gains of 6.4%, 13.6%, and 14.5% over Llama 2, CodeLlama-Python, and DeepSeek-Coder-Base respectively on MMLU, BBH, and GSM8K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama 2 on harmonized multi-source data | AlchemistCoder-L · HumanEval | pass@1: 42.7% | Helped |
| Fine-tune Llama 2 on harmonized multi-source data | AlchemistCoder-L · MBPP | pass@1: 28.4% | Helped |
| Fine-tune DeepSeek-Coder-Base on harmonized multi-source data | AlchemistCoder-DS · HumanEval and MBPP | pass@1: 79.9% and 77.0%; overall improvement of 33.3% | Helped |
| Fine-tune on harmonized multi-source data | AlchemistCoder models · DS-1000 | pass@1: up to 19.2% | Helped |
| Add AlchemistPrompts to 5% of training samples (ablation) | CodeLlama-Python-7B fine-tuning · HumanEval and MBPP | pass@1 | Helped |
| Vary the AlchemistPrompt-customized data proportion from 0% to 20% (ablation) | CodeLlama-Python-7B fine-tuning · HumanEval and MBPP | pass@1: 0% to 20%; 1% to 5% | Mixed |
| Augment original samples with AlchemistPrompt-customized variants (ablation) | CodeLlama-Python-7B fine-tuning · HumanEval and MBPP | pass@1 | Helped |
| Replace original samples with AlchemistPrompt-customized variants (ablation) | CodeLlama-Python-7B fine-tuning · HumanEval and MBPP | pass@1 | Helped |
| Add code comprehension task data (ablation) | AlchemistCoder · HumanEval and MBPP | pass@1: 5.1% | Helped |
| Fine-tune with harmonized multi-source data | AlchemistCoder models · MMLU, BBH, and GSM8K | overall performance: 6.4%, 13.6%, and 14.5% | Helped |
Surprising findings
- Naively mixing multi-source code data (DirectlyMix) can lead to inferior performance due to conflicting languages and response styles.
- CodeLlama-Python performs worse than Llama 2 on general benchmarks (MMLU, BBH, GSM8K), attributed to catastrophic forgetting from code-only fine-tuning.
Limitations
- Relies on GPT-4 to generate AlchemistPrompts, incurring generation cost.
- Gains partly depend on augmenting data with GPT-3.5/GPT-4-generated content, reducing independence from proprietary models.
Experimental setup
Datasets: Evol-Instruct-Code-80k-v1 CodeExercise-Python-27k evol-codealpaca-v1 EvolCode (generated from gpt-3.5-turbo) AlchemistPrompt-customized data code comprehension task data
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ HumanEval-X DS-1000 MMLU BBH GSM8K
Metrics: pass@1
Baselines: GPT-3.5-Turbo GPT-4-Turbo CodeLlama-Instruct WizardCoder DeepSeek-Coder-Instruct Magicoder MagicoderS-CL MagicoderS-DS
Builds on: Evol-Instruct hindsight experience replay EvalPlus Magicoder
Closest papers by meaning
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.90)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.89)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.88)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.