Research map / Vision-language and multimodal / Visual question answering
Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APL, an object-oriented AVQA network with question-conditioned and modality-conditioned clue modules plus two object-aware adaptive-positivity contrastive losses, achieving state-of-the-art on MUSIC-AVQA.
Problem
Audio-visual question answering on untrimmed videos requires finding question-relevant audio-visual clues, but prior methods ignore fine-grained audible objects and do not explicitly align question-object and audio-object pairs.
Approach
Treats detected visual objects as the bridge between question and audio: question-conditioned and modality-conditioned transformer modules encode cross-modal relations, and adaptive thresholded positive pairs (question-object, audio-object) per frame are trained with contrastive losses added to cross-entropy.
Key results
- APL with DETR reaches 74.53% overall accuracy on MUSIC-AVQA, versus 72.33% for prior SOTA COCA.
- APL with Faster R-CNN reaches 73.91% overall accuracy on MUSIC-AVQA.
- Ablation: removing both QCD and MCC drops overall accuracy from 74.53% to 26.09%; using only one module gives about 71.6-71.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APL-DETR instead of COCA | MUSIC-AVQA · All question types | accuracy: 72.33 -> 74.53 | Helped |
| Use APL-DETR instead of COCA | MUSIC-AVQA · A Question | accuracy: 75.42 -> 78.09 | Helped |
| Use APL-DETR instead of COCA | MUSIC-AVQA · V Question | accuracy: 75.23 -> 79.69 | Helped |
| Use APL-DETR instead of COCA | MUSIC-AVQA · A-V Question | accuracy: 69.96 -> 70.96 | Helped |
| Add QCD to a network without QCD or MCC (ablation) | APL · MUSIC-AVQA (All) | accuracy: 26.09 -> 71.94 | Helped |
| Add MCC to a network without QCD or MCC (ablation) | APL · MUSIC-AVQA (All) | accuracy: 26.09 -> 71.64 | Helped |
| Use both QCD and MCC rather than QCD only (ablation) | APL · MUSIC-AVQA (All) | accuracy: 71.94 -> 74.53 | Helped |
| Use both QCD and MCC rather than MCC only (ablation) | APL · MUSIC-AVQA (All) | accuracy: 71.64 -> 74.53 | Helped |
| Add question-object contrastive loss L_qo to cross-entropy training (ablation) | APL · MUSIC-AVQA (V Question) | accuracy: 78.52% to 80.47% | Helped |
| Add audio-object contrastive loss L_ao to cross-entropy training (ablation) | APL · MUSIC-AVQA (A-V Question) | accuracy: 69.64% to 70.43% | Helped |
| Set positivity threshold φ to 0 instead of 0.011 (ablation) | APL-DETR · MUSIC-AVQA (All) | accuracy: 74.53 -> 73.39 | Hurt |
| Use DETR rather than Faster R-CNN for object detection (ablation) | APL · MUSIC-AVQA (All) | accuracy: 73.91 -> 74.53 | Helped |
Surprising findings
- Removing both QCD and MCC modules collapses accuracy to 26.09%, far below using either module alone.
Limitations
- Performance is sensitive to the positivity threshold, which must be tuned relative to the number of detected objects N.
- Setting the threshold to zero (all objects positive) gives sub-optimal results.
- Evaluated only on MUSIC-AVQA.
Experimental setup
Datasets: MUSIC-AVQA
Benchmarks: MUSIC-AVQA
Metrics: QA accuracy average accuracy per question type
Baselines: AVSD Pano-AVQA STG COCA
Builds on: MUSIC-AVQA Transformer DETR Faster R-CNN VGGish
Closest papers by meaning
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering (NeurIPS 2024 · similarity 0.92)
- Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning (AAAI 2025 · similarity 0.91)
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.90)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.88)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.88)
- Patch-level Sounding Object Tracking for Audio-Visual Question Answering (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.