Research map / Vision-language and multimodal / Cross-modal learning
Parameter Efficient Multimodal Transformers for Video Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Mid fusion instead of Early fusion (ablation) | audio-visual Transformer · Kinetics-Sounds audio-visual | top-1 accuracy: 64.9 -> 65.7 | Helped |
| use Mid fusion instead of Late fusion (ablation) | audio-visual Transformer · Kinetics-Sounds audio-visual | top-1 accuracy: 61.0 -> 65.7 | Helped |
| add CPP to Late fusion (ablation) | Late fusion · Kinetics-Sounds audio-visual | top-1 accuracy: 60.6 -> 61.0 | Helped |
| sample similar rather than dissimilar content-aware negatives (ablation) | MEP pretraining · Kinetics-Sounds | top-1 accuracy: 66.2 -> 67.5 | Helped |
| use current mini-batch negatives instead of current-sequence negatives (ablation) | MEP pretraining · Kinetics-Sounds | top-1 accuracy: 64.6 -> 65.5 | Helped |
| share only the low-rank U component across Transformers (ablation) | Multi-6 · Kinetics-Sounds | top-1 accuracy: 65.7 -> 67.5 | Helped |
| share Transformer parameters across layers (ablation) | visual Transformer · Kinetics-Sounds | top-1 accuracy: 43.8 -> 43.5 | No clear effect |
| remove the CPP pretraining task (ablation) | Mid fusion · Kinetics-Sounds | top-1 accuracy: 67.5% -> 65.9% | Hurt |
| remove the MEP pretraining task (ablation) | Mid fusion · Kinetics-Sounds | top-1 accuracy: 67.5% -> 64.6% | Hurt |
| remove the multimodal Transformer MEP loss term (ablation) | Mid fusion · Kinetics-Sounds | top-1 accuracy: 67.5% -> 66.7% | Hurt |
| remove the CPP term using randomly sampled position embeddings (ablation) | Mid fusion · Kinetics-Sounds | top-1 accuracy: 67.5% -> 66.9% | Hurt |
| add modality embeddings to the multimodal Transformer (ablation) | Mid fusion · Kinetics-Sounds | top-1 accuracy: 67.1% -> 67.5% | Helped |
Closest papers by meaning
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.89)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.88)
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities (CVPR 2024 · similarity 0.88)
- A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter (AAAI 2025 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.