Research map / Large language models / Instruction and fine-tuning
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fine-tuning suppresses pretrained capabilities by shifting implicit task inference, and proposes conjugate prompting (e.g., translating prompts to non-English languages) to partially recover them in synthetic linear regression and LLMs.
Problem
Fine-tuning (instruction tuning, code fine-tuning, safety RLHF) causes catastrophic forgetting on tasks outside the fine-tuning distribution; the paper asks whether these capabilities are lost or merely suppressed.
Approach
Hypothesizes that LMs implicitly infer the task from the prompt and fine-tuning shifts this task inference rather than erasing capabilities; conjugate prompting transforms the prompt to a lower-likelihood-under-fine-tuning form (e.g., scaling labels, translating to other languages) whose answer can be inverted back to the original task.
Key results
- Translating ICL prompts reduces the drop in ICL accuracy for Alpaca from 56.75% (English) to 29.00% (French) and 1.00-1.50% for Leetspeak.
- On XNLI, Code LLaMa loses 8.36% accuracy in English versus LLaMa-2 but changes by under 2% in French, Spanish, and German, with Spanish slightly improved.
- On AdvBench, gpt-3.5-turbo answer frequency falls from 92% to 3% in English but only from 71% to 65% in Malayalam.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction-tune LLaMa to Alpaca and prompt in English | LLaMa · ICL vs instruction following | ICL accuracy: 92.00 % -> 35.25 % | Hurt |
| Translate Alpaca prompts from English to French | Alpaca · ICL vs instruction following | ICL accuracy: 35.25 % -> 69.50 % | Helped |
| Translate Alpaca prompts from English to Leetspeak | Alpaca · ICL vs instruction following | ICL accuracy: 35.25 % -> 75.00 % | Helped |
| Instruction-tune LLaMa-2 on code and evaluate in English | LLaMa-2 · XNLI, English | accuracy: 44.26 % -> 35.90 % | Hurt |
| Code fine-tune LLaMa-2 and prompt in French | LLaMa-2 · XNLI, French | accuracy: 33.53 % -> 34.98 % | Helped |
| Code fine-tune LLaMa-2 and prompt in Spanish | LLaMa-2 · XNLI, Spanish | accuracy: 38.11 % -> 38.88 % | Helped |
| Code fine-tune LLaMa-2 and prompt in German | LLaMa-2 · XNLI, German | accuracy: 34.50 % -> 33.49 % | Hurt |
| Safety fine-tune GPT-3.5 and prompt in English | GPT-3.5 · Harmful instruction following, English | answer frequency: 92 % -> 3 % | Hurt |
| Safety fine-tune GPT-3.5 and prompt in Malayalam | GPT-3.5 · Harmful instruction following, Malayalam | answer frequency: 71 % -> 65 % | No clear effect |
| Fine-tune the linear-regression model on discrete tasks | Mixture-pretrained transformer · Continuous weight distribution | loss | Hurt |
| Fine-tune the linear-regression model on discrete tasks | Mixture-pretrained transformer · Discrete weight distribution | loss | Helped |
| Apply conjugate prompting with scaled labels | Fine-tuned linear-regression transformer · Continuous weight distribution | loss | Helped |
Surprising findings
- Fine-tuning on a discrete distribution causes a sharp rise in continuous loss, and the model's loss on the forgotten task sometimes improves with lower prompt likelihood under the fine-tuning distribution.
- Code LLaMa slightly increases accuracy on Spanish and French XNLI after code fine-tuning.
Limitations
- Translation is imperfect due to third-party services, low-resource languages, and contextual knowledge, which can violate invertibility.
- Conjugate prompting requires knowledge of training data and deployment tasks.
- Mechanistic claims about task inference are untested; the framework is an assumption used to interpret results.
Experimental setup
Datasets: XNLI MNLI AdvBench Synthetic linear regression tasks
Benchmarks: XNLI AdvBench In-context learning vs instruction following templates
Metrics: loss on continuous and discrete distributions first-word ICL accuracy XNLI accuracy answer frequency vs refusal
Baselines: Alpaca Vicuna-7b OPT-IML-1.3b Code LLaMa gpt-3.5-turbo
Builds on: in-context learning of linear functions Bayes-optimal implicit inference PPO-ptx
Closest papers by meaning
- On the Loss of Context Awareness in General Instruction Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.89)
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs (ICML 2025 · similarity 0.89)
- When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations (ICLR 2024 · similarity 0.88)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.88)
- Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.