AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Everything at Once - Multi-Modal Fusion Transformer for Video Retrieval

Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio S. Feris, David Harwath, James Glass, Hilde Kuehne

CVPR 2022 · Vision-language and multimodal › Cross-modal learning · approach: Multi-modality image fusion · 143 citations (OpenAlex)

Check your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use fused video-audio input rather than averaging independent video and audio embeddings (ablation)Ours with R152 + RX101 · YouCook2 zero-shot retrievalR@10: 49.2 -> 51.3Improved
Use fused video-audio input rather than averaging independent video and audio embeddings (ablation)Ours with R152 + RX101 · MSR-VTT zero-shot retrievalR@10: 33.2 -> 31.8Hurt
Add a transformer to process modalities individually (ablation)No-transformer model · YouCook2 zero-shot retrievalR@10: 41.4 -> 50.7Improved
Replace separate modality transformers with shared fusion transformer, without fused-modality loss (ablation)Single-modality-transformer model · YouCook2 zero-shot retrievalR@10: 50.7 -> 50.2Hurt
Add combinatorial loss and fused-modality training to the fusion transformer (ablation)Fusion transformer · YouCook2 zero-shot retrievalR@10: 50.2 -> 51.3Improved
Use modality-specific final projections instead of a shared final projection (ablation)Ours · YouCook2 zero-shot retrievalR@10: 48.7 -> 51.3Improved
Remove the [cls] token and use modality-specific final projections (ablation)Ours · YouCook2 zero-shot retrievalR@10: 46.9 -> 51.3Improved
Omit positional embeddings (ablation)Ours with uniform long-video sampling · YouCook2 zero-shot retrievalR@10: 43.4 -> 45.1Improved
Process the full long video at once instead of uniformly sampling features (ablation)Ours without positional embeddings · YouCook2 zero-shot retrievalR@10: 45.1 -> 51.3Improved
Use RoPE instead of omitting positional embeddings (ablation)Ours · YouCook2 zero-shot retrievalR@10: 51.2 vs 51.3No clear effect
Increase the transformer from one block to two and reduce clips per video (ablation)Ours · YouCook2 zero-shot retrievalR@10: 51.3 -> 47.6Hurt
Temporally shift audio to disentangle it from ASR text (ablation)Ours without loss weighting · YouCook2 zero-shot retrievalR@10: 47.2 -> 45.7Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy