Research map / Large language models / Alignment and preferences
GRATH: Gradual Self-Truthifying for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRATH, a self-supervised post-processing method that generates pairwise correct/incorrect answers to out-of-domain ARC-Challenge questions via few-shot prompting and iteratively refines them with DPO (LoRA) to boost TruthfulQA truthfulness of 7B LLMs.
Problem
LLMs often generate untruthful content, with modest TruthfulQA MC1/MC2 accuracy; existing truthfulness fine-tuning needs in-domain, human-annotated answers, and DPO can be vulnerable to domain shift.
Approach
Use the model itself to generate correct and incorrect answer pairs for out-of-domain questions via few-shot prompting with TruthfulQA demonstrations, fine-tune with DPO, then iteratively regenerate the correct answers using the improved model and repeat (gradual self-truthifying), without any annotated answers.
Key results
- GRATH on Llama2-Chat-7B reaches TruthfulQA MC1 54.71% and MC2 69.10%, up from 30.23% and 45.32%, exceeding 70B open models such as Xwin-LM v0.1 (40.27% MC1).
- GRATH on Zephyr improves TruthfulQA MC1 from 42.23% to 53.86% and MC2 from 57.83% to 66.73%.
- Standard DPO on the same ARC-C data improves Llama2-Chat-7B MC1 to 36.72% and MC2 to 52.13%, below GRATH; SFT degrades truthfulness (MC1 22.40%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply GRATH to Llama2-Chat-7B using ARC-Challenge-sourced self-generated pairwise truthfulness data and one gradual refinement iteration | Llama2-Chat-7B · TruthfulQA MC1 | accuracy: 30.23 -> 54.71 | Helped |
| Apply GRATH to Llama2-Chat-7B using ARC-Challenge-sourced self-generated pairwise truthfulness data and one gradual refinement iteration | Llama2-Chat-7B · TruthfulQA MC2 | accuracy: 45.32 -> 69.10 | Helped |
| Apply GRATH | Zephyr-7B · TruthfulQA MC1 | accuracy: 42.23 -> 53.86 | Helped |
| Apply GRATH | Zephyr-7B · TruthfulQA MC2 | accuracy: 57.83 -> 66.73 | Helped |
| Apply GRATH | Llama2-Chat-7B · MMLU | accuracy: 48.14 -> 46.88 | Mixed |
| Apply DPO using OOD ARC-Challenge annotated pairwise answers instead of GRATH's self-generated pairwise data | Llama2-Chat-7B · TruthfulQA MC1 | accuracy: 30.23 -> 36.72 | Helped |
| Use OOD DPO instead of the pretrained model | Zephyr-7B · TruthfulQA MC1 | accuracy: -4.2% | Hurt |
| Use fewer pairwise truthfulness training examples at fixed DPO executions (ablation) | GRATH · TruthfulQA MC1 | accuracy | Hurt |
| Increase the number of DPO executions beyond the effective early iterations (ablation) | GRATH · TruthfulQA MC1 | accuracy | Mixed |
| Use the pretrained model rather than the current base model as DPO's reference across iterations (ablation) | GRATH · TruthfulQA MC1 | accuracy | Hurt |
| Increase the domain gap in training answers using stronger Shakespearean-style transformations (ablation) | DPO self-truthifying · TruthfulQA MC1 | accuracy | Hurt |
| Use in-domain TruthfulQA questions rather than OOD questions (ablation) | DPO self-truthifying · TruthfulQA | accuracy | Helped |
Surprising findings
- Many self-generated 'correct' answers are actually ground-truth incorrect, suggesting the gains come from relative differences between answer pairs rather than absolute correctness.
- Extra iterations beyond about one (T=1) degrade performance across benchmarks and MC2 becomes NaN from the third iteration, attributed to DPO overfitting.
Limitations
- Overfitting with repeated DPO iterations causes degraded performance and NaN MC2 scores beyond a few iterations.
- Generated 'correct' answers are often ground-truth incorrect, so the training signal is noisy and relies on relative differences.
- Evaluation is limited to TruthfulQA and a few Open LLM Leaderboard benchmarks; results focus on 7B models with some 13B results in the appendix.
Experimental setup
Datasets: ARC-Challenge TruthfulQA HellaSwag MMLU HaluEval
Benchmarks: TruthfulQA ARC-Challenge HellaSwag MMLU
Metrics: MC1 accuracy MC2 accuracy normalized accuracy
Baselines: SFT DPO RLHF RL RepE ITI Xwin-LM v0.1 Mistral-Instruct v0.1 Zephyr Llama2-Chat
Builds on: DPO RepE ITI TRL
Closest papers by meaning
- Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension (ICML 2024 · similarity 0.88)
- HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.87)
- Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning (AAAI 2024 · similarity 0.87)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.87)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
- Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.