Research map / Large language models / Instruction and fine-tuning
Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoIF, which uses LLM-generated verification code and unit tests with execution-feedback rejection sampling to automatically build instruction-following data for SFT and offline/online DPO.
Problem
Automatically constructing high-quality instruction-following training data for LLMs without manual annotation or reliance on proprietary model distillation.
Approach
Transform verification of instruction-following data quality into code verification: LLMs generate verifiable instructions, Python verification functions and unit tests, filtered by cross-validation and back-translation with an NLI model; execution feedback then drives rejection sampling for SFT and DPO pairs.
Key results
- Qwen2-72B-Instruct with online DPO reaches 88.0 instruction-level loose accuracy on IFEval; LLaMA3-70B with online DPO reaches 90.4, the first reported above 90%.
- Qwen2-7B SFT + online DPO on Qwen2-72B data improves IFEval prompt (loose) from 43.6 to 46.6 and FollowBench average SSR from 52.3 to 56.6.
- With GPT-4 as supervision model, Qwen2-7B SFT + online DPO reaches 59.5 prompt-level loose IFEval, a gain of over 15 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply AutoIF self-alignment with Online DPO | Qwen2-72B-Instruct · IFEval instruction loose | accuracy: +1.1 | Helped |
| Apply AutoIF self-alignment with Online DPO | LLaMA3-70B-Instruct · IFEval instruction loose | accuracy: +1.6 | Helped |
| Apply AutoIF strong-to-weak training with Online DPO | Qwen2-7B · IFEval prompt loose | accuracy: 43.6 -> 46.6 | Helped |
| Use Online DPO instead of Offline DPO | Qwen2-7B AutoIF · FollowBench SSR avg | SSR: 54.0 -> 56.6 | Helped |
| Replace Qwen2-72B with GPT-4 as supervision model (ablation) | Qwen2-7B AutoIF SFT · IFEval prompt loose | accuracy: 44.5 -> 52.9 | Helped |
| Add back-translation verification (ablation) | Qwen2-7B AutoIF with Online DPO · IFEval instruction loose | accuracy: +1.7 | Helped |
| Add query quality verification (ablation) | Qwen2-7B AutoIF with Online DPO · IFEval instruction loose | accuracy: +2.4 | Helped |
| Add instruction cross-verification (ablation) | Qwen2-7B AutoIF with Online DPO · IFEval instruction loose | accuracy: +3.0 | Helped |
| Add instruction- and query-level quality filtering (ablation) | Qwen2-7B AutoIF with Online DPO · IFEval instruction loose | accuracy: +3.8 | Helped |
| Increase the pass-rate threshold for instruction data (ablation) | AutoIF SFT data · IFEval | accuracy | Helped |
| Increase the pass-rate threshold for query data beyond 0.8 (ablation) | AutoIF SFT data · IFEval | accuracy | Mixed |
Surprising findings
- Supervision model quality strongly affects the result: switching the data generator from Qwen2-72B to GPT-4 raised IFEval loose prompt accuracy by over 15 points.
- The authors note that removing cross-verification of instructions causes the largest performance drop among the filtering components.
Limitations
- Focuses on single atomic constraints; cross-instruction (compositional) construction is left to future work.
- Yield is low: pass rates of about 26-34% after quality control, and the approach depends on the supervision model's coding ability.
- Relies on a stronger model for best results; LLaMA3-70B and Qwen2-72B data give smaller gains than GPT-4 data.
Experimental setup
Datasets: ShareGPT IFEval-style seed instructions (handwritten) Self-generated AutoIF instructions
Benchmarks: IFEval FollowBench C-Eval MMLU GSM8K HumanEval
Metrics: prompt-level strict and loose accuracy instruction-level strict and loose accuracy SSR (satisfaction rate) per level accuracy pass@1
Baselines: Qwen2-7B LLaMA3-8B Qwen2-72B-Instruct LLaMA3-70B-Instruct Mixtral-8x22B GPT-4 GPT-3.5 Turbo ShareGPT SFT
Builds on: Self-Instruct Direct Preference Optimization rejection sampling fine-tuning
Closest papers by meaning
- SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models (ICLR 2025 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.90)
- VERSE: Verification-based Self-Play for Code Instructions (AAAI 2025 · similarity 0.90)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.90)
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.89)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.