Research map / Vision-language and multimodal / Multimodal LLMs
Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VTI, a training-free test-time latent steering method that shifts vision-encoder and text-decoder hidden states along precomputed directions to reduce object hallucination in LVLMs.
Problem
Hallucination in large vision-language models, where generated text contains inaccurate descriptions of the image, with the hypothesis that vision feature instability and text-decoder sensitivity to vision inputs drive it.
Approach
Precompute a visual direction from the difference between averaged embeddings over random-masked image copies and original embeddings (first principal component over ~50 examples), and a textual direction from hallucinated vs. non-hallucinated caption pairs; add them to hidden states at inference to stabilize vision features and reduce text-only bias, with no training or extra forward passes.
Key results
- On POPE, VTI reaches average accuracy 86.5 on LLaVA-1.5 versus 79.8 for vanilla decoding, and outperforms VCD and OPERA on all three LVLMs.
- On CHAIR with LLaVA1.5, VTI lowers CHAIR_S from 51.0 (vanilla) to 35.8 and CHAIR_I from 15.2 to 11.1 while keeping recall around 76.8.
- On MMHAL-Bench with LLaVA-1.5, VTI achieves average score 2.90 and hallucination rate 0.51, versus 1.99 and 0.62 for vanilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply visual and textual latent-space intervention (VTI) | LLaVA-1.5 · POPE | accuracy: 84.2 -> 86.5 | Helped |
| apply visual and textual latent-space intervention (VTI) | InstructBLIP · POPE | F1 score: 81.0 -> 83.2 | Helped |
| apply visual and textual latent-space intervention (VTI) | Qwen-VL · POPE | accuracy: 84.5 -> 85.2 | Helped |
| apply visual and textual intervention | LLaVA-1.5 · CHAIR | CHAIR_S: 51.0 -> 35.8 | Helped |
| apply visual and textual intervention | LLaVA-1.5 · CHAIR | Recall: 75.2 -> 76.8 | Hurt |
| add visual intervention (ablation) | LLaVA-1.5 VTI · CHAIR | CHAIR_I: 12.9 -> 11.1 | Helped |
| add textual intervention (ablation) | LLaVA-1.5 VTI · CHAIR | CHAIR_S: 43.2 -> 35.8 | Helped |
| apply visual intervention only (ablation) | LLaVA-1.5 · CHAIR | CHAIR_I: 15.2 -> 12.7 | Helped |
| apply textual intervention only (ablation) | LLaVA-1.5 · CHAIR | CHAIR_S: 51.0 -> 41.0 | Helped |
| apply visual and textual intervention | LLaVA-1.5 · MMHAL-Bench | Average score: 2.69 -> 2.90 | Helped |
| apply visual and textual intervention | LLaVA-1.5 · MMHAL-Bench | Hallucination rate: 0.56 -> 0.51 | Helped |
Surprising findings
- Adding random noise alone tends to increase hallucination, so the reduction comes from the averaging process rather than the noise itself.
- Simple averaging of perturbed vision features drops linear-probe classification accuracy to about 62%, which motivates the softer intervention.
Limitations
- Hallucination analysis and evaluation are restricted mainly to object hallucination on POPE, CHAIR and MMHAL-Bench.
- Steering strengths alpha and beta are grid-searched per setting, with different values for CHAIR and other experiments.
Experimental setup
Datasets: POPE MSCOCO MMHAL-Bench
Benchmarks: POPE CHAIR MMHAL-Bench
Metrics: accuracy F1 score CHAIR_S CHAIR_I recall hallucination rate average score
Baselines: Vanilla decoding VCD OPERA DOLA
Builds on: representation engineering contrastive decoding (VCD) OPERA
Closest papers by meaning
- Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.95)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.95)
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.95)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.95)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.94)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.