Research map / Vision-language and multimodal / Cross-modal learning
Siamese Vision Transformers are Scalable Audio-visual Learners
Yan-Bo Lin, Gedas Bertasius
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| share one ViT between audio and visual instead of separate encoders (ablation) | AVSiam-Large · AudioSet-2M | A+V mAP: 52.0 -> 52.1 with 640M -> 332M params, 20.6G -> 10.9G memory | No clear effect |
| use multi-ratio masking instead of best fixed-ratio masking (ablation) | AVSiam-Base · AudioSet-20K | mAP: 40.8 -> 41.3 mAP; 362 -> 160 V100 hours | Improved |
| use multi-ratio masking instead of no-masking contrastive learning (ablation) | AVSiam-Base · AudioSet-20K | mAP: 40.4 -> 41.3 mAP; 510 -> 160 V100 hours | Improved |
| use higher fixed masking ratio (75%/75%) (ablation) | AVSiam-Base · AudioSet-20K | mAP: 40.8 -> 38.6 mAP | Hurt |
| use AST instead of ViT as shared encoder (ablation) | AVSiam · AudioSet-20K / VGGSound | visual-only and A+V: audio 36.5 -> 37.5; visual 23.7 -> 9.6; A+V 41.6 -> 38.1 (AS-20K) | Mixed |
| add unsupervised audio-visual contrastive pretraining (ablation) | AVSiam-Base · AudioSet-20K / VGGSound | mAP / accuracy: 39.9 -> 41.3 mAP; 61.2 -> 64.4 acc | Improved |
| add MAE reconstruction loss to contrastive pretraining (ablation) | AVSiam-Base · AudioSet-20K / VGGSound | mAP / accuracy: 41.3 -> 41.6 mAP; 64.4 -> 64.9 acc; +17 V100 hours | Improved |
| scale pretraining data (add VGGSound and ACAV) | AVSiam-Base · AS-20K / AS-2M / VGGSound | mAP / accuracy: +1.4, +1.3, +1.8 | Improved |
| scale backbone from Base to Huge | AVSiam · AS-20K / AS-2M / VGGSound | mAP / accuracy: +3.4 mAP, +4.1 mAP, +3.1% | Improved |
| use shared encoder instead of separate encoders on audio-only and visual-only inputs | AVSiam vs CAV-MAE · AudioSet-2M visual-only | mAP: 11.1 -> 30.8 | Improved |
Closest papers by meaning
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.90)
- EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning (ICML 2024 · similarity 0.88)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.88)
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.88)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.88)
- Revisiting Audio-Visual Segmentation with Vision-Centric Transformer (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
