Research map / Large language models / Instruction and fine-tuning
Teaching LLMs How to Learn with Contextual Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes contextual fine-tuning (CFT), which prepends educational-style prompts to domain text during continued training of Llama-2 models, and shows gains over continued pretraining and AdaptLLM on medical and financial benchmarks.
Problem
Updating LLMs with new domain knowledge via fine-tuning often causes forgetting or loss of instruction-following, and standard continued pretraining is inefficient at improving reasoning and recall in domain tasks.
Approach
Generalize instruction tuning into contextual fine-tuning: prepend a randomly sampled cognitively motivated contextual prompt to each domain text and train the model to predict the text tokens conditioned on that prompt, so the prompt shapes gradients toward useful semantic structure.
Key results
- On medical benchmarks, Llama-2-7B Chat with CFT averages 47.81% accuracy vs 45.96% for CPT and 45.05% for the base Chat model.
- On financial benchmarks, Llama-2-7B Chat with CFT reaches 67.96 average F1 vs 63.64 for CPT.
- CFT at 7B averages 4.89% higher than AdaptLLM on the medical benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use contextual fine-tuning instead of continued pretraining | Llama 2 7B Chat · Medical MMLU + MedQA (average) | accuracy: 45.96 -> 47.81; 1.85% | Helped |
| Use contextual fine-tuning instead of continued pretraining | Llama 2 13B Chat · Medical MMLU + MedQA (average) | accuracy: 54.31 -> 56.56; 2.25% | Helped |
| Use contextual fine-tuning instead of continued pretraining | Llama 2 7B Chat · FiQA + Causal20 + MultiFin (average) | F1: 4.32% | Helped |
| Use contextual fine-tuning instead of continued pretraining | Llama 2 13B Chat · FiQA + Causal20 + MultiFin (average) | F1: 3% | Helped |
| Replace contextual prompts with misleading negative prompts (ablation) | Llama 2 7B Chat (CFT) · Medical benchmarks | accuracy: -2.57% | Hurt |
| Replace contextual prompts with misleading negative prompts (ablation) | Llama 2 13B Chat (CFT) · Medical benchmarks | accuracy: -1.36% | Hurt |
| Replace contextual prompts with misleading negative prompts (ablation) | Llama 2 7B Chat (CFT) · Financial benchmarks | F1: -3.41% | Hurt |
| Replace contextual prompts with misleading negative prompts (ablation) | Llama 2 13B Chat (CFT) · Financial benchmarks | F1: -2.39% | Hurt |
| Use text-adaptive generated prompts instead of manually designed contextual prompts (ablation) | Llama 2 7B Chat · Medical MMLU + MedQA (average) | accuracy: 45.05 -> 46.31; 45.96 -> 46.31 | Helped |
| Use CFT + IFT instead of CPT + IFT (ablation) | Llama 2 7B Base · Medical benchmarks | accuracy: 2.95%; 1.91% | Helped |
| Use CFT + IFT instead of CPT + IFT (ablation) | Llama 2 7B Base · Financial benchmarks | F1: 36.28%; 8.67% | Helped |
| Use CFT rather than AdaptLLM domain-specific continued pretraining | Llama 2 7B · Medical benchmarks | accuracy: 4.89% on average | Helped |
Surprising findings
- Negative contextual prompts (misleading instructions) still outperform CPT on financial benchmarks, though they reduce CFT performance.
- Text-adaptive prompts generated by GPT-4o-mini yield 46.31% average medical accuracy, above the Chat and CPT baselines but below manually designed CFT prompts.
Limitations
- Gains are modest (about 1-3 points on medical benchmarks) and the paper does not aim for state-of-the-art.
- OpenMedText textbook text was OCR-extracted and cleaned with ChatGPT, which may introduce errors; MDPI articles have peer-review quality concerns and source bias.
- Evaluation limited to Llama-2 models at 7B and 13B and to a narrow set of medical and financial tasks.
Experimental setup
Datasets: OpenMedText financial news articles (306,242) FinAlpaca UltraChat MMLU medical subsets MedQA FiQA MultiFin Causal20
Benchmarks: MMLU medical subsets (Anatomy, Clinical Knowledge, College Biology, College Medicine, Medical Genetics, Professional Med MedQA FiQA Causal20 MultiFin
Metrics: accuracy F1 average accuracy relative improvement %Δ
Baselines: Continued pretraining (CPT) Instruction fine-tuning (IFT) AdaptLLM Negative contextual fine-tuning
Builds on: instruction tuning continued pretraining chain-of-thought prompting
Closest papers by meaning
- Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study (NeurIPS 2024 · similarity 0.89)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.88)
- On the Loss of Context Awareness in General Instruction Fine-tuning (NeurIPS 2025 · similarity 0.88)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.