Research map / Large language models / Language, safety and interpretability
Self-Alignment with Instruction Backtranslation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes instruction backtranslation: a LLaMA seed model predicts instructions for web text (self-augmentation), keeps only high-quality pairs via self-scoring (self-curation), and iteratively finetunes to produce the Humpback model, which tops the non-distilled Alpaca leaderboard.
Problem
Building high-quality instruction-following LLMs requires large amounts of human-annotated instructions or distillation from stronger models, which is hard to scale; the paper asks whether a model can improve itself using unlabelled web text.
Approach
Treat human-written web segments as gold responses and have a backward model generate their instructions (backtranslation), then use the same seed model to score and select the highest-quality pairs, iterating the augment-curate-finetune loop; curation is shown to be essential.
Key results
- Humpback 65B reaches an 83.71% win rate on AlpacaEval and Humpback 33B 79.84%, the top non-distilled models on the Alpaca leaderboard at their scales.
- Training on self-augmented data without self-curation fails to improve instruction following, while curated data (score >= 5) yields steady gains with scale.
- Humpback 65B improves zero-shot commonsense and MMLU accuracy over LLaMA 65B (MMLU 59.0 vs 54.8 zero-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply instruction backtranslation to LLaMA 33B | LLaMA 33B · SIQA | zero-shot accuracy: 50.2 -> 53.4 | Helped |
| Apply instruction backtranslation to LLaMA 33B | LLaMA 33B · PIQA | zero-shot accuracy: 82.2 -> 74.5 | Hurt |
| Apply instruction backtranslation to LLaMA 33B | LLaMA 33B · Arc-Challenge | zero-shot accuracy: 54.8 -> 68.5 | Helped |
| Apply instruction backtranslation to LLaMA 33B | LLaMA 33B · OBQA | zero-shot accuracy: 58.6 -> 46.4 | Hurt |
| Apply instruction backtranslation to LLaMA 65B | LLaMA 65B · MMLU | zero-shot accuracy: 54.8 -> 59.0 | Helped |
| Add self-curated augmented data to seed data (ablation) | LLaMA instruction tuning · Instruction-following evaluation | win rate against text-davinci-003 | Helped |
| Add self-curation to self-augmented data (ablation) | LLaMA instruction tuning · Instruction-following evaluation | win rate against text-davinci-003 | Helped |
| Train on uncurated augmented data without seed data (ablation) | LLaMA instruction tuning · Instruction-following evaluation | win rate against text-davinci-003 | Hurt |
| Use separate seed/web system prompts in training and their combined prompt at inference (ablation) | LLaMA instruction tuning · System prompt evaluation | win rate: 59.96 ±3.09 -> 66.47 ±3.04 | Helped |
| Use the seed system prompt at inference instead of the combined prompt (ablation) | LLaMA instruction tuning · System prompt evaluation | win rate: 66.47 ±3.04 -> 62.69 ±3.06 | Hurt |
| Remove the system prompt at inference while retaining separate training prompts (ablation) | LLaMA instruction tuning · System prompt evaluation | win rate: 66.47 ±3.04 -> 62.70 ±3.07 | Hurt |
| Increase self-curated augmented-data quantity (ablation) | 7B LLaMA instruction tuning · Instruction-following evaluation | win rate against text-davinci-003 | Helped |
Surprising findings
- Training on uncurated self-augmented web data does not improve, or even deteriorates, instruction following, showing curation is critical.
- Intermediate self-curation models have low precision (0.44-0.52) yet training on their selected data still improves instruction following.
Limitations
- Relies on GPT-4 judgments (AlpacaEval) and a small human evaluation for quality claims.
- Web-sourced augmented data may amplify biases; the model's safety was only checked on 30 prompts.
- Self-curation precision is modest, so many selected examples may still be noisy.
Experimental setup
Datasets: Open Assistant (3200 first-turn rank-0 seed examples) ClueWeb English (502k sampled segments) Vicuna Self-instruct Koala HH_RLHF LIMA test prompts AlpacaEval CrowS-Pairs
Benchmarks: AlpacaEval SIQA PIQA Arc-Easy Arc-Challenge OpenBookQA MMLU CrowS-Pairs
Metrics: win rate vs text-davinci-003 accuracy precision recall scaling coefficient alpha
Baselines: text-davinci-003 LIMA Guanaco OASST SFT OASST RLHF Vicuna WizardLM Alpaca-GPT4 LLaMA 2 Chat 70B Falcon Instruct
Builds on: backtranslation self-instruct LIMA superficial alignment LLaMA
Closest papers by meaning
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.89)
- From Instance Training to Instruction Learning: Task Adapters Generation from Instructions (NeurIPS 2024 · similarity 0.88)
- Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels (NeurIPS 2024 · similarity 0.88)
- LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions (NeurIPS 2025 · similarity 0.88)
- Toward Verifiable Instruction-Following Alignment for Retrieval Augmented Generation (AAAI 2025 · similarity 0.88)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.