Research map / Large language models / Alignment and preferences
FLAME : Factuality-Aware Alignment for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that standard SFT and DPO alignment can increase hallucination and proposes FLAME, which uses self-generated responses for fact-based instructions in SFT and a factuality reward model for DPO preference pairs, improving FActScore while keeping instruction following.
Problem
Conventional LLM alignment (SFT followed by RL/DPO) fails to improve factual accuracy and often increases hallucination, because human-written responses introduce unfamiliar knowledge and helpfulness rewards favor long, detailed responses.
Approach
Classify instructions as fact-based; for fact-based ones, train SFT on the model's own few-shot generations (not human responses) and build DPO pairs with a retrieval-augmented atomic-fact factuality reward, while keeping a self-rewarding instruction-following reward for other pairs.
Key results
- FLAME (SFT+FA-DPO variant, FA-SFT + FA-DPO) raises FActScore on Biography from 44.7 (SFT) to 47.9 and to 49.5 for FA-SFT alone, with a 51.2% Alpaca Eval win rate over the SFT+DPO baseline, per the paper's text.
- The standard SFT+DPO baseline reduces Biography FActScore relative to pre-trained 5-shot (53.1 vs 44.7 for SFT), indicating human-data SFT encourages hallucination.
- Factuality-aware DPO on FA-SFT reaches 58.9 FActScore on FAVA vs 55.0 for FA-SFT + DPO without factuality pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use factuality-aware SFT and combine instruction-following with factuality preference optimization | Llama-2 70B · Alpaca Fact | FActScore: 41.4 -> 48.7 | Helped |
| use factuality-aware SFT and combine instruction-following with factuality preference optimization | Llama-2 70B · Alpaca Eval | win rate: 50.4 -> 51.2 | Helped |
| use human responses for non-fact-based instructions and PT generations for fact-based instructions | Llama-2 70B SFT · Bio | FActScore: 44.7 -> 49.5 | Helped |
| fine-tune using human responses rather than PT-generated responses | Llama-2 70B SFT · Alpaca Eval | win rate: 48.1 -> 50.0 | Helped |
| fine-tune using human responses rather than PT-generated responses | Llama-2 70B SFT · Alpaca Fact | FActScore: 42.0 -> 38.6 | Hurt |
| add instruction-following DPO | Llama-2 70B SFT · Bio | FActScore: 44.7 -> 42.3 | Hurt |
| add factuality-only DPO rather than instruction-following DPO (ablation) | Llama-2 70B SFT · Bio | FActScore: 42.3 -> 47.1 | Helped |
| combine instruction-following and factuality preference pairs | Llama-2 70B SFT · Alpaca Eval | win rate: 50.0 -> 51.7 | Helped |
| remove fact-based instruction classification (ablation) | factuality-aware SFT · Bio | FActScore: 48.4 -> 49.5 | Hurt |
| remove fact-based instruction classification (ablation) | SFT + factuality-aware DPO · Alpaca Eval | win rate: 46.8 -> 51.7 | Hurt |
| classify fact-based sentences and exclude other sentences from fact checking (ablation) | SFT + factuality-aware DPO · Bio | FActScore: 45.0 -> 42.9 | Hurt |
| enumerate all possible response pairs instead of selecting maximum- and minimum-reward responses (ablation) | SFT + factuality-aware DPO · Alpaca Eval | win rate: 51.7 -> 50.7 | Hurt |
Surprising findings
- SFT on more factual retrieval-augmented generations made the model less factual (FActScore 35.7 vs 37.9 for PT-generated SFT data).
- DPO using RAG outputs as positives and the model's own outputs as negatives drastically lowered FActScore (23.5), while self-generated FActScore-selected pairs improved it.
Limitations
- Only two alignment skill sets (instruction following and factuality) are considered.
- Sentence-level fact-based classification is noisy and removing non-fact sentences reduced performance.
- Possible trade-off between factuality and instruction following at large scale is unknown.
Experimental setup
Datasets: OpenAssistant oasst1 Alpaca Eval FActScore biography entities FAVA annotations TruthfulQA Wikipedia Dec. 2021 dump
Benchmarks: Alpaca Eval Biography (FActScore) Alpaca Fact FAVA TruthfulQA
Metrics: win rate FActScore number of correct/erroneous facts BLEU ROUGE
Baselines: SFT SFT + DPO Llama-2-Chat 70B FActScore-based DPO (Tian et al.) SFT + factuality-only DPO
Builds on: self-rewarding language models direct preference optimization FActScore DRAGON+
Closest papers by meaning
- Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs (ICLR 2025 · similarity 0.91)
- Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators (AAAI 2025 · similarity 0.90)
- Learning to Reason for Factuality (ICML 2026 · similarity 0.89)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.89)
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.