Research map / Vision-language and multimodal / Multimodal LLMs

One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a training-free unified framework that uses vision tokens for both augmented-image enhancement (SVC) and pruned-token latent negative calibration (CRC) to reduce object hallucination in MLLMs.

Problem

Object hallucination in multimodal LLMs, where visual signals weaken during generation and language priors dominate; existing training-free fixes either boost vision or suppress text inertia separately, and naive combinations fail.

Approach

Operate entirely on intermediate hidden representations: SVC injects attention over original plus augmented vision tokens into a middle layer, while CRC builds a hallucination direction from differences between original and randomly pruned (5-token) vision-token forward passes and subtracts it in shallow layers.

Key results

• On POPE with LLaVA-1.5, the method reaches 86.79% accuracy on MSCOCO and 82.23% on AOKVQA, versus 84.79% and 77.23% for vanilla decoding.
• On CHAIR with LLaVA-1.5 at 64 tokens, CHAIR_Sen drops to 18.1 from 25.4 for vanilla.
• Latency overhead is 1.06x over greedy decoding (32.1 ms/token), versus 2.4x for VCD (72.72 ms/token).

Surprising findings

• Naively combining PAI and VCD yields no improvement over the individual approaches, according to the authors.
• LLaVA-1.5 POPE accuracy stays stable even when only 50 of 576 vision tokens are retained, with decline only below 20 tokens.

Limitations

• Relies on hand-chosen hyperparameters (lambda_s, lambda_c, K, N_h) selected on LLaVA-1.5 POPE, with generalization to other models assessed only partially.
• MMHal-Bench evaluation uses only 96 questions and GPT-4 judging, which is a small and noisy setting.

Experimental setup

Datasets: MSCOCO AOKVQA GQA

Benchmarks: POPE CHAIR MMHal-Bench MME

Metrics: accuracy F1 CHAIR_S CHAIR_I MMHal-Bench score MME perception and cognition scores latency throughput peak memory

Baselines: Vanilla VCD PAI VISTA ONLY ICD

Builds on: visual contrastive decoding PAI VISTA ONLY

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
• Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.94)
• Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing (NeurIPS 2025 · similarity 0.94)
• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation (ICML 2026 · similarity 0.94)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.94)