Research map / Large language models / Instruction and fine-tuning
Instruction Tuning With Loss Over Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Instruction Modelling (IM), which computes the training loss over instruction/prompt tokens as well as outputs, and shows it helps instruction tuning most when instructions are long relative to outputs and when training data is small.
Problem
Standard instruction tuning computes loss only on output tokens, which may lead to overfitting, especially on small or short-output datasets; the paper asks when including instruction loss helps.
Approach
Instead of masking the instruction/prompt tokens out of the loss, train the LM with next-token loss over both instruction and completion tokens (excluding template tokens), framed as a regularizer that reduces overfitting in low-resource or SAH settings.
Key results
- On AlpacaEval 1.0 with Less MMLU Chat and Less Tydiqa, IM roughly doubles or more the win rate over IT (e.g., 4.42 to 9.78 on Less MMLU Chat; 5.12 to 10.10 on Less Tydiqa).
- Mean NLP task score on Alpagasus Dolly 3k rises from 46.58 (IT) to 48.95 (IM) with Llama-2-7B.
- IM achieves a lower mean test loss on Tulu V2 (1.17 vs 1.32 for IT), while having a higher training loss on LIMA (1.45 vs 1.37).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply loss to instruction and completion tokens instead of completion tokens only | Llama-2-7B-Base instruction tuning on Alpagasus Dolly 3k · Alpagasus Dolly 3k | mean NLP score: 46.58 -> 48.95 | Helped |
| Apply loss to instruction and completion tokens instead of completion tokens only | Llama-2-7B-Base instruction tuning on Alpagasus Dolly 9k · Alpagasus Dolly 9k | AlpacaEval 1.0 win rate: 21.54 -> 30.77 | Helped |
| Apply loss to instruction and completion tokens instead of completion tokens only | Llama-2-7B-Base instruction tuning on Less MMLU Chat · Less MMLU Chat | AlpacaEval 1.0 win rate: 4.42 -> 9.78 | Helped |
| Apply loss to instruction and completion tokens instead of completion tokens only | Llama-2-7B-Base instruction tuning on LIMA · LIMA | AlpacaEval 1.0 win rate: 33.06 -> 32.94 | No clear effect |
| Apply instruction loss on datasets with long instructions and short outputs | Instruction Modelling · Less MMLU Chat and Less Tydiqa | AlpacaEval 1.0 win rate | Helped |
| Use fewer training examples while holding instruction-to-output length ratio around 10 | Instruction Modelling on Tulu V2 · AlpacaEval 1.0 | win rate | Helped |
| Apply instruction loss rather than output-only loss | Llama-2-7B-Base trained on LIMA · LIMA training set | output loss: 1.37 -> 1.45 | Hurt |
| Apply instruction loss rather than output-only loss | Llama-2-7B-Base evaluated on Tulu V2 sampled test data · Tulu V2 test data (10% sample) | output loss: 1.32 -> 1.17 | Helped |
| Apply instruction loss rather than output-only loss | Instruction Modelling on training prompts · LIMA, Less Tydiqa, Less MMLU Chat, Less BBH ICL, Alpagasus Alpaca 5k, Alpagasus Dolly 9k, Alpagasus Dolly 3k | BLEU overlap with training outputs: 18.15 -> 17.30; 69.21 -> 65.63; 72.43 -> 69.20; 60.96 -> 53.94; 72.26 -> 70.50; 61.76 -> 60.61; 60.99 -> 59.04 | Helped |
| Add KL divergence loss to instruction tuning (ablation) | Llama-2-7B-Base on Alpagasus Dolly 9k · Alpagasus Dolly 9k | mean NLP score: 45.54 -> 49.31 | Helped |
| Add KL divergence loss to instruction tuning (ablation) | Llama-2-7B-Base on LIMA · LIMA | AlpacaEval 2.0: 2.58 -> 0.06 | Hurt |
| Set KL loss coefficient to λ=0.01 rather than λ=1.0 (ablation) | Llama-2-7B-Base instruction tuning on LIMA · LIMA | AlpacaEval 2.0: 0.06 -> 2.58 | Helped |
Surprising findings
- IM produces higher training loss but lower test loss, and lower BLEU overlap with training outputs, indicating reduced overfitting.
- Applying KL divergence regularization largely removes the AlpacaEval 2.0 gains (e.g., LIMA 2.58 to 0.06), though it reduces NLP task degradation.
Limitations
- Gains depend on quality and diversity of instructions in the training data; poorly defined instructions may hurt.
- Improvements are context-dependent; e.g., on LIMA gains are modest and combining with NEFTune can hurt some NLP tasks.
- The paper explicitly does not propose IM as a replacement for standard fine-tuning, and benefits are mainly shown in low-resource settings.
Experimental setup
Datasets: Stanford Alpaca Dolly ShareGPT Code Alpaca Science Literature WizardLM Tulu V2 LIMA Less MMLU Chat Less BBH ICL Less Tydiqa Alpagasus Dolly 3k
Benchmarks: MMLU PIQA OpenbookQA HellaSwag LAMBADA WMT 2014 WMT 2016 WSC WinoGrande ARC CoQA GSM8K
Metrics: accuracy normalized accuracy F1 pass@1 exact match win rate BLEU MT-Bench score
Baselines: Instruction Tuning (IT) NEFTune KL divergence regularization
Builds on: instruction tuning NEFTune Superficial Alignment Hypothesis LIMA LESS
Closest papers by meaning
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.92)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.90)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.90)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.90)
- Improving Instruction-Following in Language Models through Activation Steering (ICLR 2025 · similarity 0.90)
- Enhancing NLU in Large Language Models Using Adversarial Noisy Instruction Tuning (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.