Research map / Large language models / Reasoning and chain of thought
Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that outcome-based RL reasoning fine-tuning increases hallucinations, gives a theoretical account, and proposes FSPO, which adds step-wise evidence-based factuality rewards and token-level advantage adjustment within GRPO.
Problem
Reasoning-oriented RL fine-tuning of LLMs increases hallucinations in their reasoning chains, even when final answers are sometimes correct.
Approach
FSPO checks each reasoning sentence against Wikipedia evidence with an NLI verifier (HHEM-2.1) yielding +1/0/-1 factuality scores, and flips or keeps token-level GRPO advantages accordingly so factually wrong tokens are penalized even in wrong-answer rollouts.
Key results
- FSPO (Qwen-Base) reaches 83.0 on HaluEval-QA and 58.4 on TruthfulQA, the highest among open-source and reasoning models in the table.
- FSPO (Qwen-Base) gets GSM8K 89.5% vs 73.2% for Qwen2.5-7B-Instruct and 77.5% for Llama3.1-8B-Instruct.
- FSPO (Qwen-Instruct) scores 20.0% on AIME 2024 and 13.3% on AIME 2025.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add step-wise factuality reward and factuality-aware token advantages (ablation) | GRPO on Qwen2.5-7B-Base · MATH-500 | Pass@1 accuracy | Helped |
| Add step-wise factuality reward and factuality-aware token advantages (ablation) | GRPO on Qwen2.5-7B-Base · HaluEval-QA | accuracy | Helped |
| Add step-wise factuality reward and factuality-aware token advantages | GRPO on Qwen2.5-7B-Base · HaluEval-QA | accuracy: 48.0 -> 83.0 | Helped |
| Add step-wise factuality reward and factuality-aware token advantages | Qwen2.5-7B-Base · GSM8K | Pass@1 accuracy: 65.2 -> 89.5 | Helped |
| Add step-wise factuality reward and factuality-aware token advantages | Qwen2.5-7B-Base · TruthfulQA | truthful output ratio: 38.2 -> 58.4 | Helped |
| Add step-wise factuality reward and factuality-aware token advantages | Qwen2.5-7B-Base · HalluQA | truthful output ratio: 39.5 -> 52.0 | Helped |
| Train using 8K rather than 1K–4K HotpotQA samples (ablation) | FSPO on Qwen2.5-7B-Base · MATH-500 | accuracy | Hurt |
| Increase HotpotQA training samples to 4K or 8K rather than 2K (ablation) | FSPO on Qwen2.5-7B-Base · HaluEval-QA | accuracy | Hurt |
| Use FSPO with Reinforce++ instead of Reinforce++ alone (ablation) | Reinforce++ on Qwen2.5-7B-Base · MATH-500 | Pass@1 accuracy | Helped |
| Use FSPO with Reinforce++ instead of Reinforce++ alone (ablation) | Reinforce++ on Qwen2.5-7B-Base · HaluEval-QA | accuracy | Helped |
Surprising findings
- Reasoning-tuned models such as R1-Distill-Qwen-7B score far below their base models on hallucination benchmarks (e.g., HalluQA 3.1 vs 32.0 for Qwen2.5-7B-Instruct).
- Adding factuality reward alone to GRPO was less effective than the full advantage-adjusted FSPO, per the ablation.
Limitations
- Theory restricted to binary rewards and simplified assumptions.
- Experiments limited to 7B/8B models due to compute; no 14B or 32B training.
- Factuality verification depends on Wikipedia evidence, so applicability is mainly to knowledge-intensive tasks.
Experimental setup
Datasets: HotpotQA 2WikiMultiHopQA SimpleRL TruthfulQA HaluEval-QA HalluQA GSM8K MATH-500 AIME 2024 AIME 2025
Benchmarks: GSM8K MATH-500 AIME 2024 AIME 2025 TruthfulQA HaluEval-QA HalluQA
Metrics: Pass@1 accuracy truthful rate discrimination accuracy factuality score
Baselines: GRPO GRPO w/ factuality REINFORCE++ DeepSeek-R1 DeepSeek-V3 QwQ-32B R1-Distill-Qwen-7B Qwen2.5-7B-Instruct Llama3.1-8B-Instruct
Builds on: GRPO PPO REINFORCE++ DeepSeek-R1 SimpleRL
Closest papers by meaning
- Learning to Reason for Hallucination Span Detection (ICLR 2026 · similarity 0.92)
- Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models (CVPR 2026 · similarity 0.92)
- Learning to Reason for Factuality (ICML 2026 · similarity 0.91)
- Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization (AAAI 2026 · similarity 0.90)
- BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation (NeurIPS 2026 · similarity 0.90)
- FLAME : Factuality-Aware Alignment for Large Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.