Research map / Large language models / Reasoning and chain of thought

Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Theory of autoregressive reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Token-Guard, a training-free decoding method that scores candidate tokens and segments with latent-space hallucination scores, locally regenerates weak fragments, and globally re-selects reasoning chains to reduce hallucinations on HALU QA benchmarks.

Problem

LLMs hallucinate content inconsistent with the input; existing decoding methods lack explicit token-level hallucination checking, quantified hallucination risk, and dynamic correction, while RAG and RLHF are resource-intensive.

Approach

A multi-stage decoding pipeline that combines hidden-state similarity to the accepted context with token probability for token-level scoring, segment-level scoring via weighted hidden states plus consistency and alignment, local window regeneration, and a global chain-level factual and logical score with iterative re-generation.

Key results

• Token-Guard averages 51.03 F1 with Llama-3.1-8B-Instruct versus 34.73 for the best baseline (Guided Decoding) in the main table, and 53.98 F1 with Qwen3-8B versus 45.79 for Guided Decoding.
• On the StrategyQA evaluation with LLaMA-7B, Token-Guard reaches 69.4% accuracy versus 64.1% for DoLa.
• Ablation on DROP_history and RAGTruth shows removing token-level scoring drops average F1 from 56.23 to 37.31.

Surprising findings

• Predictive Decoding collapses to near-zero F1 on several datasets with Qwen3-8B (average 23.67 F1 versus a 44.25 base model).
• EM scores are near zero on RAGTruth and PubMedQA for all methods, so the paper relies on F1 for these tasks.

Limitations

• Relies on heuristic hyperparameters chosen by grid search, with thresholds and weights set manually.
• Gains are limited on knowledge-intensive tasks such as PubMedQA and CovidQA, where the method cannot supply missing domain knowledge.
• Several reported numbers are inconsistent across tables (e.g., 3B/13B results and the main-table average), and theoretical propositions are informal.

Experimental setup

Datasets: FinanceBench DROP (history, nfl) CovidQA PubMedQA HaluEval (qa_samples) RAGTruth StrategyQA

Benchmarks: FinanceBench RAGTruth CovidQA DROP_history DROP_nfl PubMedQA HaluEval StrategyQA

Metrics: EM F1 BLEU time per sample tokens/sec peak memory

Baselines: BaseModel Chain-of-Thoughts Tree-of-Thought Guided Decoding Predictive Decoding Self-RAG SELF-REFINE Self-Reflection DoLa

Builds on: Chain-of-Thought Tree-of-Thought Guided Decoding Predictive Decoding DoLa

Closest papers by meaning

• BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation (NeurIPS 2026 · similarity 0.94)
• Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.93)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.91)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.91)
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.91)
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (CVPR 2024 · similarity 0.91)