Research map / Vision-language and multimodal / Visual question answering
Question-Aware Gaussian Experts for Audio-Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QA-TIGER, an AVQA model that injects question information early into audio-visual encoding and models question-relevant temporal segments with a Mixture of Gaussian experts over frames.
Problem
Audio-visual question answering needs question-specific temporal grounding, but existing methods use question information only at the final reasoning stage and rely on uniform or discrete Top-K frame sampling that misses fine-grained temporal cues.
Approach
Combine question-aware cross-attention fusion throughout the audio and visual encoders with a Mixture of Experts where each expert is a learnable Gaussian temporal mask whose centers are spread along the timeline, giving soft, continuous, question-conditioned frame weighting for both audio and visual streams.
Key results
- Achieves 77.62% overall accuracy on MUSIC-AVQA test, versus 76.79% for prior best TSPM.
- Reaches 67.99% overall accuracy on MUSIC-AVQA-R, versus 66.30% for TSPM.
- On MUSIC-AVQA-v2.0 biased test, 76.93% average with bias training, above LAVISH's 74.59%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QA-TIGER instead of TSPM | AVQA · MUSIC-AVQA test | accuracy: 76.79 -> 77.62 | Helped |
| Use QA-TIGER instead of TSPM | AVQA · MUSIC-AVQA-R test | accuracy: 66.30 -> 67.99 | Helped |
| Use QA-TIGER instead of the strongest listed baseline | AVQA · MUSIC-AVQA-v2.0 bias test, bias training | accuracy: 74.59 -> 76.93 | Helped |
| Use QA-TIGER instead of the strongest listed baseline | AVQA · MUSIC-AVQA-v2.0 bias test, balanced training | accuracy: 75.45 -> 76.57 | Helped |
| Use QA-TIGER instead of the strongest listed baseline | AVQA · MUSIC-AVQA-v2.0 balanced test, bias training | accuracy: 70.39 -> 73.91 | Helped |
| Use QA-TIGER instead of LAST-Att | AVQA · MUSIC-AVQA-v2.0 balanced test, balanced training | accuracy: 75.44 -> 76.43 | Helped |
| Add Question-Aware Fusion to the uniform-sampling baseline (ablation) | QA-TIGER · MUSIC-AVQA | accuracy: 75.04 -> 76.53 | Helped |
| Add Gaussian Experts to the uniform-sampling baseline (ablation) | QA-TIGER · MUSIC-AVQA | accuracy: 75.04 -> 76.05 | Helped |
| Combine Question-Aware Fusion with Gaussian Experts (ablation) | QA-TIGER · MUSIC-AVQA | accuracy: 76.53 -> 77.62 | Helped |
| Use question-relevant Top-K frame selection instead of uniform sampling (ablation) | AVQA with Question-Aware Fusion · MUSIC-AVQA | accuracy | Helped |
| Add dynamic weighting to Gaussian temporal aggregation (ablation) | AVQA with Question-Aware Fusion · MUSIC-AVQA | accuracy | No clear effect |
| Use disjoint Gaussian-center constraints (ablation) | Weighted-Gaussian temporal aggregation · MUSIC-AVQA | accuracy | Helped |
Surprising findings
- Uniform sampling and single Gaussian masks perform similarly, so gains come mainly from disjoint multi-expert Gaussians and question-aware fusion.
Limitations
- Expert count is fixed at seven and not adaptively chosen per question type.
- Improvements on AV-Counting and some categories are modest or mixed (e.g., AV-Temporal lower than TSPM).
Experimental setup
Datasets: MUSIC-AVQA MUSIC-AVQA-R MUSIC-AVQA-v2.0
Benchmarks: MUSIC-AVQA MUSIC-AVQA-R MUSIC-AVQA-v2.0
Metrics: accuracy
Baselines: ST-AVQA COCA PSTP-Net LAVISH APL TSPM LAST-Att AVSD Pano-AVQA
Builds on: Mixture of Experts TSPM PSTP-Net LAVISH ST-AVQA
Closest papers by meaning
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.90)
- Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (AAAI 2024 · similarity 0.90)
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering (NeurIPS 2024 · similarity 0.89)
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks (NeurIPS 2025 · similarity 0.89)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.88)
- Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.