Research map / Large language models / Language, safety and interpretability
Conditional Language Learning with Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes conditional finetuning, which prepends a context to documents during causal LM domain finetuning and masks the loss on context tokens, so the model learns p(x|c) rather than corpus-level statistics like topic priors, reducing forgetting.
Problem
Domain finetuning of language models with causal language modeling unselectively learns corpus statistics and biases (e.g., topic priors), causing over-adaptation and forgetting of general knowledge.
Approach
Prepend a context (domain hint, random UUID, or learned soft prompt) to each training document and compute loss only on the document tokens, so the context 'explains away' corpus-level statistics and the model learns p(x|c) instead of p(c); the context is dropped at inference.
Key results
- On continual medical textbook learning, conditional finetuning with learned context lowers average forgetting for LLaMA-2 7B from 2.5 (standard) to 2.1 and raises cumulative accuracy from 60.3 to 60.7.
- KL-divergence on C4 for LLaMA-2 7B drops from 0.082 (standard finetune) to 0.036 (conditional finetune with domain hint).
- Conditional finetuning achieves a better performance-forgetting tradeoff than standard finetuning on anatomy and SQuAD closed-book, with the tradeoff largely insensitive to context type.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add conditional finetuning with a domain-hint context | LLaMA-2 finetuning · Medical textbooks | gradient norm: 1.93 -> 1.13; 1.56 -> 1.06 | Helped |
| Add conditional finetuning with a domain-hint context | LLaMA-2 finetuning · C4 | KL-divergence to pretrained model: 0.082 -> 0.036; 0.116 -> 0.071 | Helped |
| Add conditional finetuning rather than standard finetuning | LLaMA-2 continual finetuning · Medical textbooks | Average Forgetting: 2.5 -> 2.3; 2.6 -> 2.5 | Helped |
| Add conditional finetuning rather than standard finetuning | LLaMA-2 continual finetuning · MRQA (closed-book) | Average Forgetting: 0.026 -> 0.022; 0.014 -> 0.014 | Mixed |
| Add conditional finetuning rather than standard finetuning | LLaMA-2 continual finetuning · MRQA (closed-book) | Cumulative Accuracy: 0.390 -> 0.382; 0.449 -> 0.450 | Mixed |
| Use learned soft-prompt contexts instead of random UUID contexts (ablation) | Conditional finetuning · Medical textbooks | Average Forgetting: 2.3 -> 2.1; 2.6 -> 2.3 | Helped |
| Use learned soft-prompt contexts instead of random UUID contexts (ablation) | Conditional finetuning · Medical textbooks | Cumulative Accuracy: 60.1 -> 60.7; 65.2 -> 65.6 | Mixed |
| Use learned soft-prompt contexts instead of random UUID contexts (ablation) | Conditional finetuning · MRQA (closed-book) | Average Forgetting: 0.022 -> 0.019; 0.014 -> 0.015 | Mixed |
| Use learned soft-prompt contexts instead of random UUID contexts (ablation) | Conditional finetuning · MRQA (closed-book) | Cumulative Accuracy: 0.382 -> 0.395; 0.450 -> 0.450 | Helped |
| Use any context type during conditional finetuning (ablation) | LLaMA-2 7B · C4 | loss change | Helped |
| Add conditional finetuning instead of standard finetuning | LLaMA-2 · Anatomy and SQuAD (closed-book) | QA performance at similar forgetting | Helped |
| Add conditional finetuning instead of standard finetuning | LLaMA-2 · Anatomy and SQuAD (closed-book) | maximum QA performance | No clear effect |
Surprising findings
- Random UUID contexts, which are uninformative, still lead the model to learn a conditional topic prior that behaves like a domain hint after finetuning.
- In continual learning, the choice of context matters: more dissimilar per-corpus contexts (learned soft prompts) give the best results, while similar contexts hurt.
Limitations
- Experiments use a limited-size corpus (1-10MB textbooks) and do not verify scaling to billions of tokens.
- Evaluation covers memorization/basic understanding via QA, not complex reasoning applications.
- Conditional finetuning is not presented as a solution to catastrophic forgetting; other factors also cause forgetting.
Experimental setup
Datasets: MedQA medical textbooks MRQA SQuAD NewsQA TriviaQA SearchQA HotpotQA Natural Questions C4
Benchmarks: MedQA textbook anatomy quiz (GPT-4 generated, MMLU-format) SQuAD closed-book MRQA closed-book C4 perplexity
Metrics: accuracy normalized F1 perplexity KL-divergence gradient norm average forgetting cumulative accuracy
Baselines: standard finetuning pretrained LLaMA-2
Builds on: prompt tuning causal language modeling MedQA MRQA
Closest papers by meaning
- Enhancing Elusive Clues in Knowledge Learning by Contrasting Attention of Language Models (AAAI 2025 · similarity 0.87)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.86)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.86)
- Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge (NeurIPS 2024 · similarity 0.85)
- Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.