Research map / Reinforcement learning / RL from feedback

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BALTO, a token-level RL method for RAG hallucination mitigation that projects claim-level verification onto tokens and assigns balanced positive/negative advantages within each response.

Problem

Response-level faithfulness rewards in RL penalize supported content in responses with localized hallucinations, causing spurious penalties, verbosity/length biases, and noisy gradients.

Approach

Extract checkable claims, verify them against the reference, localize erroneous tokens, and assign -1 advantage to hallucinated tokens and N-/N+ positive compensation to faithful factual tokens so the per-response advantage sums to zero, optimized with a GRPO-style clipped objective.

Key results

• BALTO reaches 97.5% faithfulness on ConFiQA with Qwen3-8B vs 94.8% for the strongest baseline (FSPO), and Q-Score 93.8.
• On FinLLM-Eval with Qwen3-8B, BALTO achieves Q-Score 53.6 vs 46.9 for GRPO_D.
• Removing positive advantages for faithful tokens (PA=0) drops ConFiQA Q-Score from 93.8 to 84.2.

Surprising findings

• Penalty-only token-level credit (PA=0) degrades faithfulness substantially, indicating positive compensation is necessary.
• FSPO attains higher informativeness on some datasets but at lower faithfulness.

Limitations

• Performance bounded by the accuracy of the LLM-based faithfulness evaluator used for claim extraction and token projection.
• Designed for retrieval-grounded settings with reference documents; open-domain or subjective tasks not explored.
• Theoretical results rely on simplifying assumptions (independent token gradients, Bernoulli rewards, N- << N+).

Experimental setup

Datasets: ConFiQA RAGTruth FinLLM-Eval

Benchmarks: ConFiQA RAGTruth FinLLM-Eval

Metrics: faithfulness informativeness Q-Score accuracy win rate

Baselines: SFT DPO GRPO (binary reward) GRPO (dense reward) FSPO RLFH-style

Builds on: GRPO FSPO RLFH DAPO DPO

Closest papers by meaning

• Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding (ICLR 2026 · similarity 0.94)
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.92)
• Learning to Reason for Hallucination Span Detection (ICLR 2026 · similarity 0.92)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.91)
• IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination (ICML 2026 · similarity 0.91)
• InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration (AAAI 2026 · similarity 0.91)