Research map / Large language models / Alignment and preferences
Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mask-DPO, a sentence-level masked variant of DPO that uses a fine-grained hallucination annotator (ANAH-v2) to learn only from factually correct sentences in preferred responses and penalize only incorrect sentences in dispreferred ones, improving factuality of Llama3.1-8B-Instruct.
Problem
Response-level factuality preference learning (DPO) penalizes truthful sentences in dispreferred responses and reinforces hallucinated sentences in preferred responses, since hallucinations are mixed with correct content, reducing factuality alignment effectiveness.
Approach
Use sentence-level factuality labels from a hallucination annotator as masks in the DPO objective so that only correct sentences of preferred samples and only incorrect sentences of dispreferred samples contribute to the loss; also studies data scaling along topics vs. questions and proposes a model-specific knowledge-graph hypothesis.
Key results
- Mask-DPO raises Llama3.1-8B-Instruct ANAH-v2 score from 49.19% to 77.53%, above Llama3.1-70B-Instruct (53.44%) and vanilla DPO (68.44%).
- On out-of-domain Biography, FActScore improves from 30.29% to 39.39%, close to Llama3.1-70B-Instruct (40.47%).
- Scaling number of topics is more effective than scaling questions per topic at equal preference-pair counts (e.g., +5.97 vs +4.50 at full scale).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sentence-level factuality masking to DPO (ablation) | Llama3.1-8B-Instruct · ANAH test | ANAH-v2 factuality score (%): 68.44 -> 77.53 | Helped |
| add sentence-level factuality masking to DPO (ablation) | Llama3.1-8B-Instruct · ANAH test | FactScore factuality score (%): 23.43 -> 25.56 | Helped |
| sample preference data from the policy model instead of InternLM2-7B-Chat-SFT (ablation) | Mask-DPO · ANAH test | ANAH-v2 factuality score (%): 59.43 -> 77.53 | Helped |
| use different contexts for preferred and non-preferred samples, adding reference documents only to preferred prompts (ablation) | Mask-DPO · ANAH test | ANAH-v2 factuality score (%): 49.19 -> 39.43 | Hurt |
| increase training topic count from 894 to 1788 (ablation) | Mask-DPO · ANAH test | ANAH-v2 factuality score (%): 65.04 -> 71.56 (+6.52) | Helped |
| increase training topic count from 1788 to 2682 (ablation) | Mask-DPO · ANAH test | ANAH-v2 factuality score (%): 71.56 -> 77.53 (+5.97) | Helped |
| increase questions per topic from 1 to 2 (ablation) | Mask-DPO · ANAH test | ANAH-v2 factuality score (%): 69.03 -> 73.03 (+4.00) | Helped |
| increase questions per topic from 2 to 3 (ablation) | Mask-DPO · ANAH test | ANAH-v2 factuality score (%): 73.03 -> 77.53 (+4.50) | Helped |
| select near rather than far training topics using Llama3.1-8B embeddings (ablation) | Mask-DPO with Llama3.1-8B · ANAH test | ANAH-v2 factuality score (%): 68.94 -> 73.43 | Helped |
| apply SFT to the base model before Mask-DPO (ablation) | Llama3.1-8B · ANAH test | ANAH-v2 score: 76.67 vs 77.53 | No clear effect |
| apply Mask-DPO instead of vanilla DPO | Llama3.1-8B-Instruct · ANAH test | ANAH-v2 score: 68.44 -> 77.53 | Helped |
| apply Mask-DPO alignment | Llama3.1-8B-Instruct · Biography test | FactScore factuality score (%): 30.29 -> 39.39 | Helped |
Surprising findings
- Preference pairs built with the reference document in the prompt for preferred samples only reduced the factuality score below the base model (39.43%).
- Sampling preference data from a non-policy model (InternLM2-7B) gave much smaller gains than sampling from the policy model.
Limitations
- Preference data is constructed and evaluated with the same annotator (ANAH-v2), creating reward hacking risk; FActScore used as partial check.
- Evaluated only on ANAH and Biography, and mainly on Llama3.1-8B-Instruct and InternLM2-7B-Chat-SFT.
- Sentence-level annotation is coarser than clause-level; finer annotators are lacking.
Experimental setup
Datasets: ANAH-v2 Biography
Benchmarks: ANAH-v2 test set Biography (FActScore) GSM8K MATH GPQA HumanEval MBPP
Metrics: factuality score (% correct sentences) FActScore best-of-N factuality perplexity
Baselines: DPO FactTune Flame TDPO DPOP SePO Llama3.1-70B-Instruct Qwen2-72B Gemma2-27B Yi1.5-34B
Builds on: DPO FactTune ANAH FActScore
Closest papers by meaning
- FLAME : Factuality-Aware Alignment for Large Language Models (NeurIPS 2024 · similarity 0.91)
- FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.88)
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (ICLR 2024 · similarity 0.88)
- Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.88)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)
- NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.