Research map / Vision-language and multimodal / Cross-modal learning
Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-visual Language Models
Ami Baid, Zihui Xue, Kristen Grauman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ACPO instead of base Video-LLaMA2 | Video-LLaMA2-7B-AV · AVHBench audio hallucination | accuracy: 74.0 -> 79.9 | Improved |
| apply ACPO | Video-LLaMA2-7B-AV · CMM Overreliance on Vision | accuracy on no-instances: 64.5 -> 82.0 | Improved |
| apply ACPO | Video-LLaMA2-7B-AV · audio captioning (original) | CIDEr x100: 27.9 -> 43.6 | Improved |
| apply ACPO | Video-LLaMA2-7B-AV · audio captioning (swap) | CIDEr x100: 13.1 -> 30.6 | Improved |
| apply ACPO | Video-LLaMA2-7B-AV · video captioning (original) | METEOR: 22.7 -> 20.4 | Hurt |
| apply ACPO to other backbones | video-SALMONN-2 · AVHBench audio hallucination | accuracy: 63.2 -> 68.8 | Improved |
| apply ACPO to Qwen2.5-Omni | Qwen2.5-Omni · AVHBench audio hallucination | accuracy: 66.7 -> 69.3 | Improved |
| add audio-attribution pairs (ablation) | ACPO · AVHBench audio hallucination | accuracy: 79.1 -> 79.9 | Improved |
| add audio-sensitivity pairs (ablation) | ACPO · AVHBench audio hallucination | F1: 79.6 -> 80.4 | Improved |
| use audio-swapped inputs for attribution pairs rather than no swap (ablation) | ACPO attribution pairs · AVHBench audio hallucination | accuracy: 76.9 -> 77.7 | Improved |
| train sensitivity pairs alone (no attribution pairs) (ablation) | ACPO · AVHBench video hallucination | accuracy | Hurt |
| use SFT on audio-swapped captions instead of preference optimization (ablation) | Video-LLaMA2-7B-AV · AVHBench video hallucination | accuracy: 77.2 -> 64.3 | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- OmniDPO — OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination (AAAI 2026) · better on 6 of 6 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.91)
- MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization (CVPR 2026 · similarity 0.90)
- SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models (CVPR 2026 · similarity 0.90)
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.90)
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.89)
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
