Research map / Large language models / Instruction and fine-tuning
On the Loss of Context Awareness in General Instruction Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that SFT with chat templates degrades context awareness (needle-in-a-haystack and contextual QA) in open LLMs, traces it to attention bias between user and assistant role tokens, and mitigates it by training with a context-dependency indicator token on attention-selected examples.
Problem
Instruction fine-tuning (SFT) with chat templates causes pre-trained LLMs to lose context awareness, i.e., the ability to retrieve and use information from user-provided context, even though pretrained versions do this well.
Approach
Attention-based metric (on a seed SFT model's context-retrieval head) scores how context-dependent each conversation turn is; context-dependent turns get a special [IND] token appended to the user instruction during SFT, so the model learns to attend to user context when the indicator is present.
Key results
- Official instruction-tuned models show higher NIH recall error than their pretrained counterparts in most families; removing chat templates recovers NIH performance (e.g., Llama-2-7b pretrained 98.94% vs Llama-2-7b-chat 92.45% without response prefix).
- Adding the [IND] indicator to fine-tuning raised Llama-2-7b NIH from 0.3378 to 0.7007 on ShareGPT and from 0.7375 to 0.9254 on WizardLM-70K, and improved SQuAD and QuAC across most settings.
- MT-Bench stayed comparable with the indicator in most cases (e.g., Llama-3-8b on WizardLM-70K 7.1125 vanilla vs 7.5250 with indicator), though Llama-2-7b on ShareGPT dropped from 6.4875 to 5.7375.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | TinyLlama-1.1B fine-tuned on ShareGPT · NIH | recall: 0.9846 -> 0.9921 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | TinyLlama-1.1B fine-tuned on ShareGPT · SQuAD | containment score: 59.73 -> 62.05 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | TinyLlama-1.1B fine-tuned on ShareGPT · QuAC | containment score: 15.50 -> 17.40 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | TinyLlama-1.1B fine-tuned on ShareGPT · DROP | containment score: 27.39 -> 27.84 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | TinyLlama-1.1B fine-tuned on ShareGPT · MT-Bench | average rating: 3.7250 -> 3.7375 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | Llama-2-7B fine-tuned on ShareGPT · SQuAD | containment score: 76.78 -> 79.09 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | Llama-2-7B fine-tuned on ShareGPT · QuAC | containment score: 23.60 -> 24.20 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | Llama-2-7B fine-tuned on ShareGPT · MT-Bench | average rating: 6.4875 -> 5.7375 | Hurt |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | Llama-3-8B fine-tuned on ShareGPT · QuAC | containment score: 24.80 -> 24.50 | Hurt |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | Llama-3-8B fine-tuned on ShareGPT · DROP | containment score: 42.15 -> 43.17 | Helped |
| Add context-dependency indicators to selected ShareGPT instructions and evaluation prompts | Llama-3-8B fine-tuned on ShareGPT · MT-Bench | average rating: 7.4375 -> 7.1625 | Hurt |
| Increase attention weights on user tokens | Llama-2-7B-Chat · NIH | recall: 0.9917 -> 0.9935 | Helped |
Surprising findings
- Context-retrieval performance of instruct models often recovers or surpasses pretrained models when chat templates are removed, suggesting the drop reflects template-induced bias rather than lost capability.
- Only a small fraction of instruction data (e.g., 4%-22% depending on dataset and threshold) is identified as context-dependent, implying context-dependent examples are sparse in common SFT datasets.
Limitations
- Only small to medium models (1.1B-8B) were fine-tuned due to compute limits; full fine-tuning and larger models not tested.
- Attention steering only applied to two Llama-2 models and degraded DROP performance, indicating side effects.
- The indicator may encode unintended stylistic differences from the selected subset, especially when that subset is small and distinct.
Experimental setup
Datasets: ShareGPT UltraChat-200k WizardLM-70K SQuAD 1.0 QuAC DROP
Benchmarks: NIH (needle-in-a-haystack) SQuAD QuAC DROP MT-Bench
Metrics: recall error (1 - recall) on NIH containment score MT-Bench average rating
Baselines: Vanilla SFT Llama-2-7b-chat Llama-3-8b-Instruct
Builds on: attention steering (Zhang et al. instruction following without fine-tuning) needle-in-a-haystack test QLoRA
Closest papers by meaning
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.91)
- Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.90)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.88)
- Teaching LLMs How to Learn with Contextual Fine-Tuning (ICLR 2025 · similarity 0.88)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.88)
- When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.