Research map / Vision-language and multimodal / Cross-modal learning
AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVCD, a training-free contrastive decoding method for audio-visual LLMs that masks attention-identified less-dominant modalities, reformulates CD for trimodal inputs, and skips CD on low-entropy steps.
Problem
Hallucinations in audio-visual LLMs arise from unimodal and cross-modal interactions among audio, video, and text, which existing single-modality contrastive decoding methods designed for VLMs do not address.
Approach
Dynamically identify less-dominant modalities from the final query token's attention, mask their high-attention tokens to create perturbed logits, and combine them in a trimodal contrastive formula (with a Taylor-approximated log-average over audio states) plus entropy-gated skipping of CD steps.
Key results
- On AVHBench, AVCD improves VideoLLaMA2 accuracy from 70.52% (Base) to 72.15%, and video-SALMONN from 58.19% to 62.18%.
- On MUSIC-AVQA, AVCD reaches 81.58% for VideoLLaMA2 vs 81.30% Base, and 49.73% for video-SALMONN vs 48.50% Base.
- Entropy-guided adaptive decoding at tau=0.8 gives faster inference than VCD (2.25 vs 2.5 sec/token) while improving accuracy over Base (78.05% to 80.98%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Base decoding with AVCD | VideoLLaMA2 · AVHBench | accuracy (%): 70.52 -> 72.15 | Helped |
| Replace Base decoding with AVCD | video-SALMONN · AVHBench | accuracy (%): 58.19 -> 62.18 | Helped |
| Replace Base decoding with AVCD | VideoLLaMA2 · MUSIC-AVQA | accuracy (%): 81.30{ $\pm$0.09} -> 81.58{ $\pm$0.03} | Helped |
| Replace Base decoding with AVCD | VideoLLaMA2 · AVHBench-Cap | Score: 2.84{ $\pm$0.01} -> 3.03{ $\pm$0.01} | Helped |
| Replace Base decoding with AVCD | VideoLLaMA2 · MSVD-QA | accuracy (%): 74.43{ $\pm$0.31} -> 75.20{ $\pm$0.42} | Helped |
| Replace Base decoding with AVCD | VideoLLaMA2 · ActivityNet-QA | accuracy (%): 47.19{ $\pm$0.55} -> 48.22{ $\pm$0.04} | Helped |
| Replace Base decoding with AVCD | Video-LLaVA · MSVD-QA | accuracy (%): 70.20{ $\pm$0.20} -> 72.16{ $\pm$0.24} | Helped |
| Replace Base decoding with AVCD | Video-LLaVA · ActivityNet-QA | accuracy (%): 47.48{ $\pm$0.02} -> 48.03{ $\pm$0.14} | Helped |
| Add dominance-aware masking (ablation) | AVCD on VideoLLaMA2 · AVHBench validation | accuracy (%): 74.15 -> 79.02 | Helped |
| Add trimodal contrastive formulation (Eq. 8) (ablation) | AVCD on VideoLLaMA2 · AVHBench validation | accuracy (%): 79.02 -> 81.95 | Helped |
| Add entropy-guided adaptive decoding (ablation) | AVCD on VideoLLaMA2 · AVHBench validation | inference speed (sec/token): 4.4 -> 3.1 | Helped |
| Set masking ratio P to 25 instead of 50 (ablation) | AVCD on VideoLLaMA2 · AVHBench validation | accuracy (%): 81.95 -> 80.98 | Hurt |
Surprising findings
- Directly extending VCD to audio degrades performance on AV-LLMs (e.g., VideoLLaMA2 MUSIC-AVQA 81.30 to 77.66; video-SALMONN 48.50 to 41.57).
- Language accounts for about 70% of final-layer attention in AV-LLMs, and the adaptive dominance choice matched fixing language as dominant on AVHBench.
Limitations
- Multiple extra forward passes increase computational cost as the number of modalities grows.
- Entropy-based skipping may miss hallucinations that occur when the model is confident.
Experimental setup
Datasets: AVHBench MUSIC-AVQA MSVD-QA ActivityNet-QA MSCOCO AOKVQA GQA OmniBench
Benchmarks: AVHBench MUSIC-AVQA AVHBench-Cap MSVD-QA ActivityNet-QA POPE OmniBench
Metrics: accuracy F1 score captioning score inference time per token
Baselines: Base decoding VCD VCD* SID OPERA CODE ICD
Builds on: visual contrastive decoding adaptive plausibility constraint DoLa SID
Closest papers by meaning
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.94)
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.93)
- ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.93)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.93)
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.93)
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.