Research map / Vision-language and multimodal / Cross-modal learning
Everything at Once - Multi-Modal Fusion Transformer for Video Retrieval
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio S. Feris, David Harwath, James Glass, Hilde Kuehne
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fused video-audio input rather than averaging independent video and audio embeddings (ablation) | Ours with R152 + RX101 · YouCook2 zero-shot retrieval | R@10: 49.2 -> 51.3 | Improved |
| Use fused video-audio input rather than averaging independent video and audio embeddings (ablation) | Ours with R152 + RX101 · MSR-VTT zero-shot retrieval | R@10: 33.2 -> 31.8 | Hurt |
| Add a transformer to process modalities individually (ablation) | No-transformer model · YouCook2 zero-shot retrieval | R@10: 41.4 -> 50.7 | Improved |
| Replace separate modality transformers with shared fusion transformer, without fused-modality loss (ablation) | Single-modality-transformer model · YouCook2 zero-shot retrieval | R@10: 50.7 -> 50.2 | Hurt |
| Add combinatorial loss and fused-modality training to the fusion transformer (ablation) | Fusion transformer · YouCook2 zero-shot retrieval | R@10: 50.2 -> 51.3 | Improved |
| Use modality-specific final projections instead of a shared final projection (ablation) | Ours · YouCook2 zero-shot retrieval | R@10: 48.7 -> 51.3 | Improved |
| Remove the [cls] token and use modality-specific final projections (ablation) | Ours · YouCook2 zero-shot retrieval | R@10: 46.9 -> 51.3 | Improved |
| Omit positional embeddings (ablation) | Ours with uniform long-video sampling · YouCook2 zero-shot retrieval | R@10: 43.4 -> 45.1 | Improved |
| Process the full long video at once instead of uniformly sampling features (ablation) | Ours without positional embeddings · YouCook2 zero-shot retrieval | R@10: 45.1 -> 51.3 | Improved |
| Use RoPE instead of omitting positional embeddings (ablation) | Ours · YouCook2 zero-shot retrieval | R@10: 51.2 vs 51.3 | No clear effect |
| Increase the transformer from one block to two and reduce clips per video (ablation) | Ours · YouCook2 zero-shot retrieval | R@10: 51.3 -> 47.6 | Hurt |
| Temporally shift audio to disentangle it from ASR text (ablation) | Ours without loss weighting · YouCook2 zero-shot retrieval | R@10: 47.2 -> 45.7 | Hurt |
Closest papers by meaning
- CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion (ICLR 2025 · similarity 0.86)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.86)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.86)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.86)
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
