Research map / Video / Video-language understanding
Multi-modal Transformer for Video Retrieval
Valentin Gabeur, Chen Sun, Karteek Alahari, Cordelia Schmid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace max-pooled expert features with MMT over all extracted features (ablation) | video encoder · MSRVTT 1k-A | R@5: 50.9 -> 54.0 | Improved |
| replace collaborative gating with MMT (ablation) | video encoder · MSRVTT 1k-A | R@5: 51.3 -> 52.5 | Improved |
| give MMT all extracted features instead of only aggregated ones (ablation) | MMT · MSRVTT 1k-A | R@5: 52.5 -> 54.0 | Improved |
| keep temporal order of features rather than shuffling (ablation) | MMT · MSRVTT 1k-A | R@5: 53.3 -> 54.0 | Improved |
| initialise F_agg with max pooling instead of zeros (ablation) | MMT · MSRVTT 1k-A | R@5: 50.2 -> 54.0 | Improved |
| initialise F_agg with max pool instead of mean pool (ablation) | MMT · MSRVTT 1k-A | R@5: 54.2 -> 54.0 | No clear effect |
| use 4 layers 4 heads instead of 2 layers 2 heads (ablation) | MMT · MSRVTT 1k-A | R@5: 53.2 -> 54.0 | Improved |
| use 8 layers 8 heads instead of 4 layers 4 heads (ablation) | MMT · MSRVTT 1k-A | R@5: 54.0 -> 53.9 | No clear effect |
| finetune BERT instead of freezing it (ablation) | caption encoder · MSRVTT (from scratch) | R@5: 17.1 -> 54.0 | Improved |
| remove stop words from captions (zero-shot) (ablation) | pretrained model · MSRVTT zero-shot | R@5: 6.9 -> 14.4 | Improved |
| remove stop words from captions (training from scratch) (ablation) | MMT · MSRVTT | R@5: 54.0 -> 50.0 | Hurt |
| remove stop words from captions (pretrain then finetune) (ablation) | MMT · MSRVTT | R@5: 57.1 -> 55.0 | Hurt |
Closest papers by meaning
- Everything at Once - Multi-Modal Fusion Transformer for Video Retrieval (CVPR 2022 · similarity 0.90)
- Thinking Fast and Slow: Efficient Text-to-Visual Retrieval With Transformers (CVPR 2021 · similarity 0.89)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.88)
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.86)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
