Research map / Vision-language and multimodal / Cross-modal learning
Vision Transformers Are Parameter-Efficient Audio-Visual Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bidirectional (A2V + V2A) cross-modal fusion versus no cross-modal connections (ablation) | LAVisH adapter · AVE | Acc: +3.2% | Helped |
| use latent tokens (compressed cross-attention) instead of direct cross-attention on all tokens (ablation) | bidirectional adapter · AVE | Acc: +1.3% (81.1 vs 79.8) | Helped |
| latent tokens instead of direct cross-attention, at ~20x lower GFLOPs for adapter modules (ablation) | LAVisH adapter · AVE computational cost | GFLOPs: 11 vs 217 GFLOPs | Helped |
| add cross-modal attention to Adapter, Compacter, LoRA baselines (ablation) | parameter-efficient baselines · AVE | Acc: Adapter 79.1 -> 79.9; LoRA 79.0 -> 79.7; Compacter 78.4 -> 78.6 | Helped |
| use audio input jointly (audio+vision) rather than vision-only (ablation) | LAVisH · AVE / AVS / AVQA | accuracy / mIoU: +5.8% AVE, +8% AVS mIoU, +13.9% AVQA | Helped |
| use Swin-V2 backbone instead of ViT backbone (ablation) | LAVisH · AVE | Acc: 81.1 vs 79.6 (ViT-L variant 78.1 vs Swin) | Helped |
| replace ViT backbone with ResNet-152 backbone (ablation) | LAVisH adapter · AVE | Acc: ~18% accuracy drop vs ViT-B | Hurt |
| add external VGGish audio features concatenated to LAVisH (ablation) | LAVisH · AVE | Acc: 81.1 -> 82.4 | Helped |
| increase number of latent tokens beyond 2 on AVE (ablation) | LAVisH · AVE | Acc: best at 2 tokens; more tokens do not improve | No clear effect |
| LAVisH frozen Swin-V2-L vs fully fine-tuned CMBS with Swin-V2-L on AVE with fewer trainable params | LAVisH vs CMBS · AVE | Acc: 81.1 vs 80.4 with 10.1M vs 14.1M trainable params | Helped |
| frozen LAVisH Swin-V2-L vs AVS fine-tuned on same backbone | LAVisH on AVSBench-S4 · AVSBench-S4 | mIoU: 80.1 vs 80.4 with 37.2M vs 249.7M trainable params | No clear effect |
| frozen LAVisH (ViT-B) vs fine-tuned MBT-B on AVE | LAVisH ViT-B · AVE | Acc: 75.3 vs 77.8 (MBT-B with 172M trainable params) | Hurt |
Closest papers by meaning
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.89)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.87)
- Revisiting Audio-Visual Segmentation with Vision-Centric Transformer (CVPR 2025 · similarity 0.87)
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.