Research map / Large language models / Instruction and fine-tuning
Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prereq-Tune, a two-stage fine-tuning method that first trains a knowledge LoRA on prerequisite facts and then trains a skill LoRA on top of it, optionally with multi-version fictitious synthetic data, to reduce hallucinations in Llama-3-8B.
Problem
Fine-tuning on data whose knowledge is unfamiliar to the pre-trained LLM encourages hallucinated outputs, due to knowledge inconsistency between pre-training and fine-tuning.
Approach
Disentangle skill and knowledge learning: first train a frozen-later knowledge LoRA on prerequisite knowledge, then train a skill LoRA on top of it so SFT learns only task skills; multi-version fictitious data with different knowledge LoRAs forces the skill LoRA to ground answers in internal knowledge, and the knowledge LoRA is dropped at inference.
Key results
- On biography generation, Prereq-Tune reaches 45.30% FActScore accuracy versus 32.70% for SFT and 41.16% for Popular, despite being trained only on fictitious data for long-form tasks.
- On medical entity generation, Prereq-Tune achieves 74.35% accuracy vs 69.94% for SFT and 70.03% for RL.
- On PopQA short QA, Prereq-Tune achieves 47.91% accuracy vs 46.42% for SFT; SFT trained on fictitious data without prerequisite learning drops to 15.44% on persons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prerequisite learning before fine-tuning on fictitious data (ablation) | SFT on fictitious data · Persons | accuracy: 15.44 -> 45.30 | Helped |
| Add prerequisite learning before fine-tuning on fictitious data (ablation) | SFT on fictitious data · Medical Entities | accuracy: 64.44 -> 74.35 | Helped |
| Add prerequisite learning before fine-tuning on fictitious data (ablation) | SFT on fictitious data · QA (PopQA) | accuracy: 44.98 -> 47.91 | Helped |
| Use both statement-based and passage-based knowledge instead of either format alone (ablation) | Prereq-Tune · QA | accuracy: 47.58 -> 47.91; 47.07 -> 47.91 | Helped |
| Use both statement-based and passage-based knowledge instead of either format alone (ablation) | Prereq-Tune · Biography generation | accuracy: 38.75 -> 45.30; 39.75 -> 45.30 | Helped |
| Use GPT-4-based atomic-claim decomposition instead of sentence-based decomposition (ablation) | Prereq-Tune · Biography generation | accuracy | Helped |
| Train with prerequisite learning on fictitious synthetic data rather than standard SFT on that data (ablation) | Fictitious-data fine-tuning · QA (original PopQA) | accuracy: 36.05 -> 37.50 | Helped |
| Train with prerequisite learning on fictitious synthetic data rather than standard SFT on that data (ablation) | Fictitious-data fine-tuning · Persons | accuracy: 15.44 -> 45.30 | Helped |
| Plug in the matching knowledge LoRA version | Prereq-Tune skill LoRA · Fictitious QA test data | accuracy: 15.52 -> 94.83 (V1); 5.17 -> 95.40 (V2) | Helped |
| Remove the knowledge LoRA at inference | Prereq-Tune skill LoRA · Fictitious training data QA | accuracy: 58.01 -> 3.99 | Hurt |
| Use the answer-abstention extension | Prereq-Tune · Known QA questions | correctly answered questions: 53.20% -> 54.94% | Helped |
| Use the answer-abstention extension | Prereq-Tune · Known QA questions | mistakenly abstained questions: 35.51% -> 33.64% | Helped |
Surprising findings
- Fine-tuning directly on fictitious synthetic data without prerequisite learning severely degrades factuality (15.44% vs 32.70% SFT on persons), while adding prerequisite learning turns the same data into a benefit.
- The skill LoRA trained with knowledge LoRAs generalizes knowledge grounding to the base model's pre-trained knowledge when the knowledge LoRA is removed, showing no measurable knowledge pollution.
Limitations
- Hallucinations are reduced but not eliminated; generated outputs still contain factual errors.
- Multi-version training is only applicable to fictitious data, since real knowledge may remain in the pre-trained model.
- Evaluation relies on FActScore with GPT-4o-mini and a modified claim filter, and results on the original PopQA differ from clarified PopQA.
Experimental setup
Datasets: PopQA Wikipedia biographies (persons, medical entities via Wikidata) Wikidata GPT-4-generated fictitious biographies and QA
Benchmarks: FActScore biography generation (183 persons) medical entity long-form generation PopQA
Metrics: FActScore accuracy (objective claims) number of claims QA accuracy memorized entities
Baselines: SFT Popular Flame FactTune RL (PPO with FActScore reward model) SFT-GPT SFT-fictitious
Builds on: LoRA FActScore FactTune Flame
Closest papers by meaning
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.87)
- Teaching LLMs How to Learn with Contextual Fine-Tuning (ICLR 2025 · similarity 0.87)
- It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.86)
- From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data (ICLR 2025 · similarity 0.86)
- Probing Hidden Knowledge Holes in Unlearned LLMs (NeurIPS 2025 · similarity 0.86)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.