Research map / Vision-language and multimodal / Multimodal LLMs
Self-Augmented Visual Contrastive Decoding
Eun Woo Im, Muhammad Kashif Ali, Vivek Gupta
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Augmented Visual Contrastive Decoding, a method that mitigates hallucination in LVLMs by building contrastive visual inputs from the text query context rather than generic visual augmentations.
Problem
LVLMs inherit hallucination tendencies from their language models; existing visual contrastive decoding uses generic visual augmentations that ignore the text query's context, limiting effectiveness.
Approach
Replaces generic visual augmentations used in visual contrastive decoding with query-aware self-augmented visual inputs, so the contrastive negative reflects the specific context of the text query.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use query-adaptive augmentation (Self-Aug) instead of a single static augmentation (ablation) | Self-Aug · MME-Perception (LLaVA-1.5-7B) | score: 1351.76 -> 1431.30 (best static noise vs Self-Aug) | Improved |
| replace APC with SAT thresholding (ablation) | Self-Aug · MME-Perception (LLaVA-1.5-7B) | score: 1345.46 -> 1431.30 | Improved |
| replace APC with SAT thresholding (ablation) | VCD · MME-Perception (LLaVA-1.5-7B) | score: 1323.67 -> 1395.17 | Improved |
| add operational knowledge (augmentation definitions) to the SAS prompt (ablation) | SAS prompting · MME-Perception (LLaVA-1.5-7B) | score: 1312.39 -> 1419.08 | Improved |
| add reasoning before the augmentation choice (ablation) | SAS prompting · MME-Perception (LLaVA-1.5-7B) | score: 1419.08 -> 1428.02 | No clear effect |
| add in-context examples to the SAS prompt (ablation) | SAS prompting · MME-Perception (LLaVA-1.5-7B) | score: 1419.08 -> 1428.63 | No clear effect |
| use inverse-entropy threshold (Hdecay) rather than direct-entropy or APC (ablation) | SAT · MME-Perception (LLaVA-1.5-7B) | score | Improved |
| use Self-Aug versus multinomial sampling on Qwen3-VL-8B | Qwen3-VL-8B · MME-Perception | score: 1725.16 -> 1726.77 | No clear effect |
| apply Self-Aug versus multinomial sampling on InstructBLIP | InstructBLIP-7B · MMVP | accuracy: 19.20 -> 16.13 | Hurt |
| use larger 13B model for augmentation selection (ablation) | Self-Aug · Agreement with Oracle | agreement: 64.15% -> 66.19% | Improved |
| use full reasoning+ICL prompt versus lightweight prompt (ablation) | Self-Aug · LLaVA-Bench (LLaVA-1.5-13B) | score: 73.82 -> 76.24 | Improved |
| use Self-Aug instead of VACoDe on POPE-MSCOCO accuracy | LLaVA-1.5-7B · POPE-MSCOCO | accuracy: 84.29 -> 82.93 | Hurt |
Experimental setup
Baselines: visual contrastive decoding
Builds on: visual contrastive decoding
Closest papers by meaning
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.89)
- ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (NeurIPS 2025 · similarity 0.88)
- Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework (CVPR 2026 · similarity 0.88)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.88)
- SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge (NeurIPS 2024 · similarity 0.88)
- Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
