Research map / Large language models / Reasoning and chain of thought

Learning to Reason for Factuality

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an online RL (GRPO) recipe with a composite reward (VeriScore factual precision, log detail level, LLM-judged relevance) to make reasoning LLMs more factual on long-form generation.

Problem

Reasoning LLMs hallucinate more than non-reasoning models on long-form factuality, and online RL for long-form factuality is hard because factuality rewards are slow to compute and can be hacked via shorter, less detailed, or irrelevant answers.

Approach

A three-part reward combining smoothed VeriScore factual precision, a log-scaled count of supported claims for detail, and a pairwise LLM-judge relevance check against the base model, plus a parallelized ~30x faster VeriScore for real-time rollouts.

Key results

• Online GRPO (SFT+GRPO) reaches 68.1% average factual precision, 23.1 points above the Llama-3.1-8B-Instruct base model, with 23% more supported claims.
• SFT+GRPO keeps a 54.4% AlpacaEval win rate vs the base model, while SFT+DPO drops to 37.8%.
• DeepSeek-R1 and QwQ-32B show 10 and 13 points lower average precision than DeepSeek-V3 and Qwen-2.5-32B, respectively.

Surprising findings

• Optimizing factual precision alone leads to reward hacking: win rate drops to 43.5% despite higher precision.
• Adding a detail reward initially lowers relevance further (win rate 36.9%) because the model inflates detail with general, less pertinent facts.

Limitations

• Relies on automatic evaluators (VeriScore, LLM judge) without human validation.
• Possible overfitting to VeriScore's specific preferences, mitigated only by using different evaluator models.
• Relevance judge is the base model itself, limiting judge strength.

Experimental setup

Datasets: WildChat LongFact

Benchmarks: LongFact FAVA AlpacaFact Biography Factory-Hard FactBench-Hard

Metrics: factual precision detail level (supported claims) win rate

Baselines: SFT SFT+DPO DeepSeek-R1 QwQ-32B DeepSeek-V3 Qwen-2.5-32B Llama-3.1-8B-Instruct

Builds on: GRPO Dr. GRPO DPO VeriScore FActScore

Closest papers by meaning

• Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models (NeurIPS 2025 · similarity 0.91)
• Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization (NeurIPS 2026 · similarity 0.90)
• FLAME : Factuality-Aware Alignment for Large Language Models (NeurIPS 2024 · similarity 0.89)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.89)
• FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.89)
• Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)