Research map / Vision-language and multimodal / Cross-modal learning
Attention Bottlenecks for Multimodal Fusion
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace late fusion with bottleneck mid-fusion | MBT audio-visual classification · Epic-Kitchens-100 | Top-1 action accuracy: 37.90 -> 43.40 | Improved |
| Replace late fusion with bottleneck mid-fusion | MBT audio-visual classification · AudioSet mini | mAP: 41.80 -> 43.92 | Improved |
| Replace vanilla cross-attention with attention bottlenecks (ablation) | AudioSet fusion model · AudioSet mini | mAP: over 2 mAP | Improved |
| Use modality-specific encoder weights rather than shared weights (ablation) | AudioSet fusion model with early fusion · AudioSet mini | mAP | Improved |
| Use modality-specific encoder weights rather than shared weights (ablation) | AudioSet fusion model with later fusion · AudioSet mini | mAP | No clear effect |
| Move fusion from early or late layers to intermediate layers (ablation) | AudioSet fusion model · AudioSet mini | mAP | Improved |
| Vary the number of bottleneck tokens from 4 to other tested sizes (ablation) | AudioSet fusion model · AudioSet mini | mAP: all within 0.5 mAP | No clear effect |
| Increase input span from 2 seconds to 8 seconds (ablation) | Audio-Visual MBT · AudioSet mini | mAP: 37.95±0.51 -> 42.37±0.44 | Improved |
| Use independent MixUp weights for each modality (ablation) | Audio-Visual MBT · AudioSet | mAP: 42.6 mAP to 43.9 mAP | Improved |
| Sample audio and video windows asynchronously rather than synchronously (ablation) | Audio-Visual MBT · AudioSet mini | mAP | No clear effect |
| Update bottlenecks sequentially rather than symmetrically (ablation) | Audio-Visual MBT · AudioSet mini | mAP: 43.66±0.26 vs 43.42±0.19; 43.23±0.12 | No clear effect |
| Scale backbone from ViT-Base to ViT-Large (ablation) | Audio-Visual MBT · AudioSet mini | mAP: 43.3 -> 42.2 | Hurt |
Closest papers by meaning
- Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · similarity 0.89)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.88)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.87)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.87)
- Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (ICLR 2024 · similarity 0.87)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
