Research map / Vision-language and multimodal / Visual question answering
AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the AVQACL continual-learning benchmark (Split-AVQA, Split-MUSIC-AVQA) and a method combining question-guided cross-modal fusion, task-specific distillation with spatial-temporal constraints, and a question semantic consistency loss.
Problem
Audio-visual question answering models assume static training data and suffer catastrophic forgetting when learning sequential AVQA tasks with new questions, audio-visual categories, and answers.
Approach
Builds continual-learning splits of AVQA and MUSIC-AVQA by question type and sound category, and proposes question-guided spatial-temporal attention fusion plus distillation with spatial-temporal feature-difference constraints and a cosine-based question semantic consistency loss on top of a replay-based setup.
Key results
- On Split-AVQA the method reaches 32.05 MA and 2.47 AF, versus 30.50 MA for SS-IL, the best baseline.
- On Split-MUSIC-AVQA the method reaches 33.64 MA and 27.08 AF, versus 32.16 MA for SS-IL.
- Ablation shows each component (QCIF, TKD-STFC, QSCC) contributes; full model is best on both datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AVQACL instead of SSIL | continual audio-visual question answering · Split-AVQA | MA: 30.50 -> 32.05 | Helped |
| Use AVQACL instead of SSIL | continual audio-visual question answering · Split-AVQA | AF: 2.68 -> 2.47 | Helped |
| Use AVQACL instead of SSIL | continual audio-visual question answering · Split-MUSIC-AVQA | MA: 32.16 -> 33.64 | Helped |
| Use AVQACL instead of SSIL | continual audio-visual question answering · Split-MUSIC-AVQA | AF: 28.05 -> 27.08 | Helped |
| Add QCIF, TKD-STFC, and QSCC (ablation) | AVQACL · Split-AVQA | MA: 18.34 -> 32.05 | Helped |
| Add QCIF, TKD-STFC, and QSCC (ablation) | AVQACL · Split-MUSIC-AVQA | MA: 28.79 -> 33.64 | Helped |
| Add QCIF, TKD-STFC, and QSCC (ablation) | AVQACL · Split-AVQA | AF: 40.94 -> 2.47 | Helped |
| Add QCIF, TKD-STFC, and QSCC (ablation) | AVQACL · Split-MUSIC-AVQA | AF: 33.17 -> 27.08 | Helped |
| Add QCIF (ablation) | AVQACL without TKD-STFC or QSCC · Split-AVQA | MA: 18.34 -> 27.97 | Helped |
| Add QSCC (ablation) | AVQACL without QCIF or TKD-STFC · Split-AVQA | MA: 18.34 -> 29.08 | Helped |
| Add TKD-STFC to QCIF (ablation) | AVQACL without QSCC · Split-AVQA | MA: 30.77 -> 32.05 | Helped |
| Add QSCC to QCIF and TKD-STFC (ablation) | AVQACL · Split-MUSIC-AVQA | MA: 31.19 -> 33.64 | Helped |
Surprising findings
- Vanilla fine-tuning and EWC have very high forgetting (about 40 AF on Split-AVQA), while the Upper Bound reaches only 55.42 MA on Split-AVQA.
Limitations
- Experiments use small traditional encoders; large multimodal models were not explored for continual AVQA.
- Memory-based methods show fluctuating performance on Split-MUSIC-AVQA as memory grows, suggesting overfitting.
- Only the two repurposed datasets are evaluated.
Experimental setup
Datasets: AVQA MUSIC-AVQA Split-AVQA Split-MUSIC-AVQA
Benchmarks: Split-AVQA Split-MUSIC-AVQA
Metrics: Mean Accuracy Average Forgetting
Baselines: Vanilla LwF EWC MAS iCaRL SS-IL AV-CIL Upper Bound
Builds on: AV-CIL iCaRL LwF Task-specific knowledge distillation
Closest papers by meaning
- Audio-Visual Adaptive Fusion Network for Question Answering Based on Contrastive Learning (AAAI 2025 · similarity 0.91)
- Question-Aware Gaussian Experts for Audio-Visual Question Answering (CVPR 2025 · similarity 0.90)
- Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering (NeurIPS 2024 · similarity 0.90)
- CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering (CVPR 2025 · similarity 0.89)
- Object-Aware Adaptive-Positivity Learning for Audio-Visual Question Answering (AAAI 2024 · similarity 0.88)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.