Research map / Vision-language and multimodal / Visual question answering
Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVAF-Net, which uses temporal and spatial contrastive alignment losses plus a question-conditioned adaptive modality-weighting module to improve audio-visual question answering on MUSIC-AVQA.
Problem
Audio-visual question answering requires aligning audio and visual cues across time and space and weighting audio versus visual modalities according to the question, which existing models handle inadequately.
Approach
Combines temporal alignment contrastive learning (TACL) and spatial alignment contrastive learning (SACL) with an audio-visual clue-mining module and a question-oriented adaptive fusion (QOAF) module that generates question-dependent modality weights.
Key results
- AVAF-Net reaches 75.90% average accuracy on MUSIC-AVQA, above all compared baselines including APL (75.72%).
- On MUSIC-AVQA-Real, AVAF-Net reaches 77.45% average accuracy versus 75.72% for APL.
- Removing all proposed modules drops average accuracy from 75.90% to 26.70% on MUSIC-AVQA; removing QOAF drops it to 72.78%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add temporal alignment contrastive learning (TACL) (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 75.35 -> 75.90 | Helped |
| add audio-visual clue mining (AVCM) (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 71.67 -> 75.90 | Helped |
| add spatial alignment contrastive learning (SACL) (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 75.50 -> 75.90 | Helped |
| add question-oriented adaptive fusion (QOAF) with question-conditioned modality weights (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 72.78 -> 75.90 | Helped |
| add all designed modules (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 26.70 -> 75.90 | Helped |
| use SACL threshold φ = 0.019 instead of other tested thresholds (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 75.50, 75.48, 75.90, 75.79, 75.54, 75.73, 75.58 | Mixed |
| use loss scaling factor β = 0.2 instead of other tested values (ablation) | AVAF-Net · MUSIC-AVQA test | average accuracy (%): 75.76, 75.90, 75.76, 75.68, 75.39, 74.87, 74.64, 74.07 | Mixed |
| use AVAF-Net instead of APL | AVQA models · MUSIC-AVQA test | average accuracy (%): 73.86 -> 75.90 | Helped |
| use AVAF-Net instead of APL | AVQA models · MUSIC-AVQA-Real test | average accuracy (%): 75.72 -> 77.45 | Helped |
Limitations
- Gains over the strongest baseline APL are small (about 0.2 points on MUSIC-AVQA), with mixed per-subtask results.
- Evaluated only on MUSIC-AVQA and its real-video subset, limiting generality to other AVQA settings.
- Hyperparameters (threshold phi, beta) were tuned on the test results reported in the paper.
Experimental setup
Datasets: MUSIC-AVQA MUSIC-AVQA-Real
Benchmarks: MUSIC-AVQA MUSIC-AVQA-Real
Metrics: accuracy
Baselines: GRU HME MCAN PASC AVSD Pano-AVQA ST-AVQA PSTP-Net QAGL APL
Builds on: PSTP-Net APL ST-AVQA CLIP VGGish
Closest papers by meaning
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering (NeurIPS 2024 · similarity 0.92)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.91)
- Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (AAAI 2024 · similarity 0.91)
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.88)
- SkillCLIP: Skill Aware Modality Fusion Visual Question Answering (Student Abstract) (AAAI 2024 · similarity 0.87)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.