Research map / Large language models / Retrieval and knowledge

Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies thinking-induced hallucination in factuality QA for Qwen3 LRMs and proposes MARGO, a GRPO variant with mixed thinking/non-thinking rollout groups where non-thinking rollouts serve as same-model advantage references.

Problem

Explicit thinking in large reasoning models improves factuality QA on average but can overturn correct direct (non-thinking) answers into hallucinations at the instance level.

Approach

Model thinking as a residual over the direct-answer tendency and build GRPO groups mixing thinking and non-thinking rollouts for the same question, so the non-thinking rollouts act as same-model references that shift thinking advantages by the residual value of thinking.

Key results

• On Qwen3-4B, MARGO averages 25.49% across six factuality benchmarks vs 22.81% for Fixed Think and 23.31% for Fixed Think+RL.
• On Qwen3-8B, MARGO averages 29.57% vs 27.59% for Fixed Think and 27.92% for Fixed Think+RL.
• On AIME24/AIME25/AMC23 with Qwen3-8B, MARGO reports Mean@16 average 80.7 vs 79.2 for Fixed Think (reported as a 1.5% gain).

Surprising findings

• Adaptive mode-selection baselines (CLS and SFT) do not consistently outperform fixed thinking mode on factuality QA.
• Standard GRPO with only thinking rollouts barely changes the harmful transition ratio r_{1,0}, suggesting RL alone does not fix thinking-induced hallucination.

Limitations

• Evaluation limited to two Qwen3 model sizes (4B and 8B) and factuality benchmarks; open-ended generation, RAG, and multimodal settings left for future work.
• Relies on Qwen3-32B LLM judge for rewards and evaluation, which may introduce judge noise.
• Training data is small (about 5.7k questions) and filtered by thinking/non-thinking gap, and a random-subset ablation was run only on Qwen3-4B.

Experimental setup

Datasets: TriviaQA training split SimpleQA SimpleQA-Verified TriviaQA NQ_Open PopQA HotpotQA AMC23 AIME24 AIME25 GSM8K

Benchmarks: SimpleQA SimpleQA-Verified TriviaQA NQ_Open PopQA HotpotQA AMC23 AIME24 AIME25

Metrics: accuracy (judge-graded) Mean@16 Pass@16 transition ratios r_{i,j} generation length

Baselines: Fixed NoThink Fixed Think Adaptive CLS (DeBERTa classifier) Adaptive SFT Adaptive RL (AdaptThink) Fixed NoThink+RL Fixed Think+RL

Builds on: GRPO DeepSeek-R1-style RLVR verl Adaptive thinking (AdaptThink)

Closest papers by meaning

• Learning to Reason for Factuality (ICML 2026 · similarity 0.90)
• Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.89)
• FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.89)
• Toward Adaptive Reasoning in Large Language Models with Thought Rollback (ICML 2024 · similarity 0.88)
• Mitigating Hallucinations in Large Language Models via Causal Reasoning (AAAI 2026 · similarity 0.88)
• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.88)