Research map / Vision-language and multimodal / Multimodal LLMs
Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CGC (GNN-based co-occurrence clustering of image tokens) and VTD (latent subtraction of visually absent dominant-cluster tokens) to reduce object hallucination in LVLMs with discrete image tokenizers.
Problem
LVLMs with discrete image tokenizers (Chameleon, Emu3, Janus-Pro) hallucinate non-existent objects; existing hallucination mitigation methods are less effective on these unified token-based models.
Approach
Hypothesizes that co-occurrence-based visual priors among discrete image tokens drive hallucination. CGC builds a token co-occurrence graph from COCO panoptic segmentation, trains a GNN with contrastive loss and clusters tokens; VTD subtracts the projection of visually absent tokens from dominant clusters from hidden states of present image tokens during decoding.
Key results
- On Janus-Pro-7B, CGC+VTD reduces Object HalBench CHAIR-s from 19.33 (Nucleus Sampling) to 10.43 and CHAIR-i from 10.68 to 5.81.
- On Chameleon-7B, CGC+VTD improves AMBER CHAIR from 19.34 to 13.88 and discriminative accuracy from 43.85 to 46.83 relative to Nucleus Sampling.
- CGC+VTD has the lowest runtime among compared mitigation methods (188 s vs 941 s for OPERA and 470 s for PROJECTAWAY) with the lowest memory (14,478 MB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CGC+VTD instead of Nucleus Sampling | Chameleon-7B · AMBER generative | CHAIR: 19.34 -> 13.88 | Helped |
| Apply CGC+VTD instead of Nucleus Sampling | Janus-Pro-7B · Object HalBench | CHAIR-s: 19.33 -> 10.43 | Helped |
| Apply CGC+VTD instead of Nucleus Sampling | Emu3-13B · Object HalBench | CHAIR-s: 23.67 -> 17.25 | Helped |
| Apply CGC+VTD instead of Nucleus Sampling | Chameleon-7B · AMBER generative | Cover: 49.43 -> 52.75 | Hurt |
| Replace CGC clustering with K-means-only clustering | CGC+VTD · AMBER and Object HalBench | hallucination metrics | Mixed |
| Combine CGC+VTD with VCD | Chameleon-7B · Object HalBench | CHAIR-s: 44.32 -> 45.10 | Hurt |
| Combine CGC+VTD with SID | Chameleon-7B · Object HalBench | CHAIR-s: 46.79 -> 42.90 | Helped |
| Combine CGC+VTD with PROJECTAWAY | Chameleon-7B · Object HalBench | CHAIR-s: 43.67 -> 39.02 | Helped |
| Combine CGC+VTD with OPERA | Chameleon-7B · Object HalBench | CHAIR-s: 39.56 -> 33.50 | Helped |
| Add semantic coherence to co-occurrence computation (ablation) | CGC+VTD on Chameleon-7B · Object HalBench | CHAIR-s: 42.15 -> 38.58 | Helped |
| Add spatial proximity to co-occurrence computation (ablation) | CGC+VTD on Chameleon-7B · Object HalBench | CHAIR-s: 43.67 -> 38.58 | Helped |
| Add positive pair similarity loss (ablation) | CGC+VTD on Chameleon-7B · Object HalBench | CHAIR-s: 41.74 -> 38.58 | Helped |
Surprising findings
- Contrastive decoding methods VCD and SID yield higher CHAIR than plain Nucleus Sampling on generative AMBER tasks for all evaluated discrete-tokenizer LVLMs.
- Cover drops slightly with CGC+VTD relative to some baselines, indicating a trade-off between hallucination reduction and object coverage.
Limitations
- Slight drop in coverage (Cover) compared with some baselines.
- Method is specific to LVLMs with discrete image tokenizers and not applicable to continuous-feature models such as LLaVA.
- Hyperparameters (layer, gamma, number of dominant clusters) are tuned per model on an AMBER subset.
Experimental setup
Datasets: COCO 2017 Panoptic validation set AMBER Object HalBench MME
Benchmarks: AMBER Object HalBench MME
Metrics: CHAIR Cover Hal Cog Accuracy F1 CHAIR-s CHAIR-i MME perception and cognition score
Baselines: Nucleus Sampling VCD SID PROJECTAWAY OPERA Cluster-Base
Builds on: PROJECTAWAY OPERA VCD SID
Closest papers by meaning
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.94)
- One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.94)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.93)
- Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow (AAAI 2025 · similarity 0.93)
- On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.