Research map / Vision-language and multimodal / Multimodal LLMs
ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Amplification Fusion (VAF), a training-free plug-and-play method that amplifies attention from instruction tokens to image tokens in middle-layer visual perception heads to reduce object hallucination without the cost of contrastive decoding.
Problem
Contrastive decoding methods such as VCD mitigate object hallucination in MLLMs but degrade content quality (CIDEr, ScienceQA) and roughly double inference time because they process extra distorted inputs.
Approach
Saliency analysis suggests middle layers (8-15) are where visual-textual fusion happens and visual attention is underweighted relative to system prompts; VAF rescales attention scores from instruction tokens to image tokens upward and system-prompt attention downward, restricted to top-50% visual perception heads, with no extra forward passes.
Key results
- On POPE (averaged over MS-COCO, A-OKVQA, GQA), VAF improves LLaVA-v1.5-7B accuracy over regular decoding by 1.8-2.5 points on random setting and up to 3.2 on Qwen-VL-7B adversarial.
- On NoCaps, VCD reduces CIDEr by 18% and ICD by 27% for LLaVA-7B, while VAF keeps CIDEr at 78.8 vs 78.7 regular.
- Inference latency on NoCaps for LLaVA-v1.5-7B: regular 0.488s, VCD 1.058s, VAF 0.489s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VAF instead of regular decoding | LLaVA-v1.5-7B · POPE | Accuracy: 87.8 -> 89.6 | Helped |
| Use VAF instead of VCD | LLaVA-v1.5-13B · POPE | Accuracy: VAF 90.1; VCD 88.9 | Helped |
| Use VAF instead of regular decoding | LLaVA-v1.5-7B · MME subset | Total Score: 610.00 -> 636.67 | Helped |
| Use VAF instead of regular decoding | LLaVA-v1.5-7B · ScienceQA | Accuracy: 68.0 -> 68.5 | Helped |
| Use VAF instead of regular decoding | LLaVA-v1.5-13B · NoCaps | CIDEr: 82.6 -> 82.3 | Hurt |
| Use VCD instead of regular decoding | LLaVA-v1.5-7B · NoCaps | CIDEr: 78.7 -> 65.7 | Hurt |
| Use VAF instead of regular decoding | LLaVA-v1.5-7B · POPE inference | Latency/Example: 0.111s -> 0.116s | No clear effect |
| Use VCD instead of regular decoding | LLaVA-v1.5-7B · POPE inference | Latency/Example: 0.111s -> 0.210s | Hurt |
| Restrict attention redistribution to visual-perception heads (ablation) | VAF on LLaVA-7B · NoCaps | CIDEr: 76.4 -> 78.8 | Helped |
| Remove visual-perception-head restriction (ablation) | VAF on LLaVA-13B · POPE | Accuracy: 90.2 -> 90.0 | No clear effect |
| Set visual enhancement coefficient α above 0.25 (ablation) | VAF on COCO-Random · POPE | hallucination suppression | Hurt |
| Enhance visual attention in middle rather than shallow or deep layers (ablation) | VAF on LLaVA-v1.5-7B · COCO-Random | hallucination suppression | Helped |
Surprising findings
- Contrastive decoding (VCD) lowers ScienceQA and NoCaps quality substantially, and ICD drops CIDEr by 27% on NoCaps for LLaVA-7B.
- Enhancing visual attention across all heads in middle layers was judged too aggressive, motivating restricting enhancement to half of the heads.
Limitations
- Hyperparameters alpha and beta are tuned on COCO-Random POPE and applied across settings; the enhancement strength shows degradation above alpha 0.25 or beta above 0.15.
- Evaluated only on three MLLMs with linear projection (LLaVA-v1.5 family and Qwen-VL), layer range fixed to 8-15 found via saliency on LLaVA-7B.
Experimental setup
Datasets: MS-COCO A-OKVQA GQA NoCaps ScienceQA MME
Benchmarks: POPE MME NoCaps ScienceQA
Metrics: accuracy F1-score CIDEr latency GPU memory
Baselines: VCD ICD Regular decoding
Builds on: Visual Contrastive Decoding Instruction Contrastive Decoding contrastive decoding
Closest papers by meaning
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.94)
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)
- Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation (ICML 2026 · similarity 0.93)
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.93)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.93)
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.