Research map / Large language models / Instruction and fine-tuning

Teaching Language Models to Hallucinate Less with Synthetic Tasks

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SynTra, which optimizes an LLM's system message via prefix-tuning on a synthetic names-retrieval task (with reference-data regularization) to reduce hallucination on realistic abstractive summarization tasks.

Problem

LLMs hallucinate on abstractive summarization tasks even when all needed information is in context, but hallucination is too expensive to evaluate at each optimization step to optimize directly.

Approach

Design a synthetic task (names retrieval) where hallucination is cheap to measure exactly, optimize a continuous postfix to the system message via prefix-tuning (or fine-tune weights) on it with a KL regularizer on reference SQuAD prompts, then transfer the learned system message to real tasks.

Key results

• Optimizing the system message with synthetic + reference data lowers Orca's average GPT-4-judged hallucination rate from 25.8% to 18.3%, including a 16-point drop on ACI-Bench.
• For Vicuna, the system message variant reduces average hallucination from 35.9% to 33.4%, while full-model fine-tuning on synthetic data alone raises it to 58.2%.
• Fine-tuning the full Orca model on synthetic data increases average hallucination from 25.8% to 31.3%.

Measured effects reported in the paper

Surprising findings

• Fine-tuning the entire model on the synthetic task can increase hallucination on realistic tasks, while optimizing only the system message reduces it.
• Training on the synthetic task alone reduces newline output rate, a spurious attribute that the reference data helps restore.

Limitations

• Gains are uneven across models: Vicuna improves much less than Orca on average (2.5 points versus 7.5).
• Hyperparameters were tuned only on Orca using MS MARCO validation data, and Vicuna was not tuned.
• Requires designing a synthetic task; the paper gives no causal account of which task properties reduce hallucination and tried only a few synthetic tasks.

Experimental setup

Datasets: MS MARCO QMSum ACI-Bench SQuAD

Benchmarks: MS MARCO QMSum ACI-Bench

Metrics: hallucination rate (GPT-4 judged) BLEU ROUGE-1 ROUGE-2 ROUGE-L ungrounded entities grounded entities exact-match correctness on names task newline rate

Baselines: Original model Full model fine-tuning on synthetic data Full model fine-tuning on synthetic + reference data

Builds on: prefix-tuning PPO-ptx LoRA

Closest papers by meaning

• ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models (NeurIPS 2024 · similarity 0.90)
• Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.90)
• LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.89)
• Attributive Reasoning for Hallucination Diagnosis of Large Language Models (AAAI 2025 · similarity 0.89)
• In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.89)
• HaDeMiF: Hallucination Detection and Mitigation in Large Language Models (ICLR 2025 · similarity 0.89)