Research map / Vision-language and multimodal / Visual question answering
Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MUSIC-AVQA-R, a robustness benchmark made by rephrasing MUSIC-AVQA test questions and splitting head/tail by answer distribution, and proposes MCCD, a plug-and-play multi-view debiasing strategy with bias learners for audio, video and question.
Problem
Existing AVQA models overlearn dataset biases and are not robust; current datasets use few templates (93-word vocabulary) and lack precise robustness diagnostics for rare vs frequent answers.
Approach
Expand the MUSIC-AVQA test split via human-verified paraphrasing and evaluate head (in-distribution) and tail (out-of-distribution) accuracy; debias with bias learners per uni-modality, enlarging distances between uni-modal and fusion logits and using KL-based cycle guidance among uni-modal logits.
Key results
- Ours reaches 66.95% overall average on MUSIC-AVQA-R versus 59.25% for LAVisH+MCCD (best other), per the robustness table in the paper's own comparison.
- On MUSIC-AVQA the method reaches 72.20% overall accuracy, beating COCA (71.64%) and STG (71.52%) under the same ResNet-18 visual encoder.
- MCCD improves 9 of 13 baselines on MUSIC-AVQA and 11 of 13 on MUSIC-AVQA-R when plugged in.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MCCD debiasing strategy | baselines (13 methods) on MUSIC-AVQA-R · MUSIC-AVQA-R test split | Avg. accuracy (All): improves on 11 of 13 baselines | Helped |
| add MCCD debiasing strategy | baselines on MUSIC-AVQA · MUSIC-AVQA test split | Avg. accuracy (All): improves on 9 of 13 baselines; drops for others (e.g., BiLSTM Attn 59.92 to 57.56) | Mixed |
| add MCCD to LAVisH | LAVisH · MUSIC-AVQA-R test split | Avg. accuracy (All): 57.63 to 59.25 | Helped |
| add MCCD to LAVisH | LAVisH · MUSIC-AVQA test split | Avg. accuracy (All): 73.76 to 73.87 | No clear effect |
| add MCCD to HCAttn/AVSD/PSAC and others | HCAttn, PSAC, HME on MUSIC-AVQA-R · MUSIC-AVQA-R test split | Avg. accuracy (All): HME 53.66 to 53.34; PSAC 50.45 to 51.31; HCAttn 51.90 to 52.29 | Mixed |
| remove multifaceted debiasing (MD) from full model (ablation) | Ours · MUSIC-AVQA-R test split | Avg. accuracy (All): 66.95 to 64.80 without MD (-2.15) | Helped |
| remove multifaceted debiasing (MD) from full model (ablation) | Ours · MUSIC-AVQA test split | Avg. accuracy (All): 72.20 to 71.48 without MD (-0.72) | Helped |
| remove cycle guidance (CG) from full model (ablation) | Ours · MUSIC-AVQA-R test split | Avg. accuracy (All): 66.95 to 65.87 without CG | Helped |
| remove cycle guidance (CG) from full model (ablation) | Ours · MUSIC-AVQA test split | Avg. accuracy (All): 72.20 to 72.45 without CG (+0.25) | Hurt |
| add audio-logit distance term d_a (ablation) | Ours · MUSIC-AVQA-R test split | Avg. accuracy (All): 66.95 vs 61.75 without d_a | Helped |
| add video-logit distance term d_v (ablation) | Ours · MUSIC-AVQA-R test split | Avg. accuracy (All): 66.95 vs 63.85 without d_v | Helped |
| add question-logit distance term d_q (ablation) | Ours · MUSIC-AVQA-R test split | Avg. accuracy (All): 66.95 vs 65.86 without d_q | Helped |
Surprising findings
- Video QA and AVQA baselines such as STG and AVSD degrade substantially on the rephrased, distribution-shifted MUSIC-AVQA-R, suggesting reliance on memorized answer statistics.
- Audio QA method CONVLSTM shows competitive robustness and the highest tail accuracy on some AVQA question types.
Limitations
- Answer space is limited to 42 single-word classes, deviating from real-world open answers.
- Model is a classifier rather than a generative model, so it is not compared with large generative backbones.
- No error bars or significance tests due to computational cost of re-evaluating 13 methods.
Experimental setup
Datasets: MUSIC-AVQA
Benchmarks: MUSIC-AVQA MUSIC-AVQA-R
Metrics: accuracy head accuracy tail accuracy overall accuracy
Baselines: FCNLSTM CONVLSTM BiLSTM Attn HCAttn MCAN GRU HME PSAC HCRN AVSD LAViT STG
Builds on: MUSIC-AVQA VisualBert VGGish LAVisH STG
Closest papers by meaning
- Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning (AAAI 2025 · similarity 0.92)
- Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (AAAI 2024 · similarity 0.92)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.90)
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.89)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.89)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.