Research map / Vision-language and multimodal / Multimodal LLMs
Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes visual adversarial perturbation (VAP), which optimizes black-box, gradient-free visual noise with CLIP-text-encoder similarity objectives to reduce object hallucination in LVMs without modifying the model.
Problem
Object hallucination in large vision-language models (LVMs), where models generate plausible but factually inaccurate object descriptions, often driven by parametric language priors and long-tail training bias.
Approach
Reframes adversarial perturbation as a corrective "cure": optimizes bounded visual noise on the input image, using zero-order gradient estimation and a CLIP text encoder as surrogate similarity measure, so that responses align with visual content and move away from responses to a noised contrastive image (conditional and unconditional settings), with no change to model weights.
Key results
- On POPE adversarial sampling, VAP improves Intern-VL2 accuracy from 82.00 to 84.81 (+2.81) and F1 by +2.09.
- On BEAF, VAP improves TU for Intern-VL2 by +2.03 and F1_TUID by +1.46 on average across models with small gains.
- On CHAIR, VAP reduces CHAIR_S for Intern-VL2 from 7.50 to 6.60 under the short-caption prompt; gains are modest (mostly under 1 point).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VAP visual noise to input images | 8 LVMs (e.g., LLaVA-v1.5, Intern-VL2) · POPE (Popular, Random, Adversarial) | Accuracy and F1: +0.34 to +2.81 accuracy | Helped |
| add VAP visual noise | 8 LVMs · BEAF | Accuracy, F1, TU, F1_TUID: TU up to +2.31; SB_n and IG degrade in some models (e.g., LLaVA-OV IG +0.18) | Mixed |
| add VAP visual noise | 8 LVMs · CHAIR caption generation | CHAIR_I and CHAIR_S: CHAIR_I down 0.06 to 0.68 points | Helped |
| replace VAP with Gaussian noise of equal strength (epsilon=2) (ablation) | 8 LVMs · POPE popular/adversarial | F1 and accuracy | Hurt |
| increase VAP perturbation strength epsilon to 16 or more (ablation) | Intern-VL2 · POPE adversarial | accuracy/F1 | Hurt |
| add VCD on top of VAP (ablation) | LLaVA-v1.5 · CHAIR | CHAIR_I: 3.82 to 3.55 (I1) | Helped |
| use smaller Intern-VL2-1B as proxy to generate perturbations for Intern-VL2-8B (ablation) | Intern-VL2-8B · POPE adversarial | F1: +1.78 F1 (80.70 to 82.52) at 1/8 inference cost | Helped |
| increase distortion level T of negative image to full Gaussian noise (ablation) | Intern-VL2 · POPE adversarial | accuracy/F1 | Hurt |
Surprising findings
- Gaussian noise of equal strength degrades performance across models, whereas VAP improves it.
- Perturbations optimized on smaller proxy models (Intern-VL2-1B) transfer to larger targets, giving a 1.78% F1 gain on Intern-VL2-8B at 1/8 the inference cost.
Limitations
- Optimizing the visual noise is computationally intensive, requiring many queries per image.
- Gains are small (often under 1-2 points) and hyperparameters (sigma, T) are tuned per model.
- Performance falls below baseline when perturbation strength epsilon >= 16, and some models (e.g., LLaVA-OV) give template-like binary answers that are hard to optimize.
Experimental setup
Datasets: MS-COCO POPE BEAF CHAIR
Benchmarks: POPE BEAF CHAIR
Metrics: accuracy precision recall F1 CHAIR_I CHAIR_S TU IG SB_p SB_n
Baselines: Gaussian noise VCD REVERIE HalluciDoctor PDM OPERA
Builds on: visual contrastive decoding zero-order optimization
Closest papers by meaning
- Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMs (AAAI 2026 · similarity 0.94)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.93)
- Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.92)
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)
- SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.