Research map / Large language models / Instruction and fine-tuning
Understanding Finetuning for Factual Knowledge Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that fine-tuning QA data about less-known facts degrades downstream factual recall in LLMs (via attention imbalance toward generic relation tokens), while fine-tuning on well-known facts helps.
Problem
Which fine-tuning QA examples improve or harm downstream factual knowledge extraction, and why does fact popularity in pretraining matter for fine-tuning outcomes?
Approach
Introduces fact salience (value-matrix association between subject and answer) and shows via a one-layer transformer that fine-tuning on less salient facts pushes attention away from subject tokens toward the QA-prompt token, suppressing stored knowledge; popular-fact fine-tuning counteracts this.
Key results
- On PopQA-Controlled, FT-Top reaches 44.5% test accuracy versus 37.4% for FT-Bottom (zero-shot 20.1%).
- On MMLU-History, Mistral-7B fine-tuned on FT-Top gets 68.7% vs 59.4% for FT-Bottom and 67.4% for FT-Whole.
- Fine-tuning on the bottom 50% of facts underperforms top 50% by about 6% on PopQA and 4% on Entity Questions, and random 50% subsets also beat FT-Bottom.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on the more popular rather than less popular half | Llama-2-7B · PopQA-Controlled test | Test-Acc: 37.4% -> 44.5 % | Helped |
| Fine-tune on the more popular rather than less popular half | Llama-2-7B · MMLU-History | accuracy: 55.6 % ($0.4$) -> 61.4% (0.3) | Helped |
| Fine-tune on the more popular rather than less popular half | Mistral-7B · MMLU-History | accuracy: 59.4% ($0.5)$ -> 68.7% ($0.5$) | Helped |
| Fine-tune on the more popular rather than less popular half | Llama-2-7B and Mistral-7B · PopQA | factual accuracy: 6% | Helped |
| Fine-tune on the more popular rather than less popular half | Llama-2-7B and Mistral-7B · Entity Questions | factual accuracy: 4% | Helped |
| Fine-tune on the most popular half rather than the whole dataset | Llama-2-7B · MMLU-History | accuracy: 58.8% $(0.2)$ -> 61.4% (0.3) | Helped |
| Fine-tune on the most popular half rather than the whole dataset | Mistral-7B · MMLU-History | accuracy: 67.4 %($0.4)$ -> 68.7% ($0.5$) | Helped |
| Fine-tune on the best top 30% of well-known facts rather than the whole dataset | MMLU models · MMLU | accuracy: up to 2.5% | Helped |
| Fine-tune on a randomly selected 50% subset rather than the less popular half | Llama-2-7B and Mistral-7B · PopQA and Entity Questions | factual accuracy | Helped |
Surprising findings
- Fine-tuning on a random or top-popularity subset can match or outperform fine-tuning on the entire dataset, so more QA data is not always better.
- Fine-tuning on fully factual but less-known facts harms downstream factuality even though all facts were seen in pretraining.
Limitations
- Theory is for a simplified one-layer, single-head transformer with fixed orthogonal embeddings.
- Popularity proxied by Wikipedia page counts or model loss rather than measured pretraining frequency.
- Real-data experiments restricted to short-answer and MMLU history subsets with exact-match evaluation.
Experimental setup
Datasets: PopQA Entity Questions MMLU (history subset) PopQA-Controlled (constructed)
Benchmarks: PopQA Entity Questions MMLU-History
Metrics: exact match accuracy accuracy by popularity percentile
Baselines: Zero-Shot Prompting FT-Whole FT-Random
Builds on: LoRA PopQA Entity Questions
Closest papers by meaning
- From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning (NeurIPS 2025 · similarity 0.89)
- Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.88)
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (ICML 2024 · similarity 0.88)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.87)
- Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts (ICML 2025 · similarity 0.86)
- Co-occurrence is not Factual Association in Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.