Research map / Video / Video-language understanding
MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuLTI, a video-language model with a Text-Guided MultiWay-Sampler (adapt-pooling plus shared self-attention sampler) for efficient fusion of long sequences, and a Multiple Choice Modeling pretraining task aligned with videoQA.
Problem
Video-and-language understanding models with heavy multimodal encoders and fusion modules are computationally expensive on long video frames and long text, and pretraining tasks do not match downstream videoQA.
Approach
Condense text features and use them to guide sampling of video features via a decoder-style sampler initialized with adapt-pooling (attention-weighted residual pooling) instead of random queries, plus a four-choice multiple-choice pretraining task built from video-text pairs.
Key results
- MuLTI-L reaches 47.8% on MSRVTT-QA and 54.7% on MSVD-QA, versus 47.4% and 52.3% for Flamingo.
- On multi-label classification, MuLTI-B with OCR reaches 67.86 mAP versus 58.53 for ALPRO (reproduced).
- MCM pretraining raises MSRVTT-QA accuracy from 46.28 to 46.61 and MSVD-QA from 51.93 to 53.03 over the pretraining baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Text-Guided MultiWay-Sampler instead of using the base fusion setup (ablation) | MuLTI-B · MSRQ | accuracy: 44.84 -> 45.54 | Helped |
| Add Text-Guided MultiWay-Sampler instead of using the base fusion setup (ablation) | MuLTI-B · MSVQ | accuracy: 48.35 -> 49.86 | Helped |
| Add Multiple Choice Modeling to the pretraining baseline (ablation) | MuLTI-B · MSRQ | accuracy: 46.28 -> 46.61 | Helped |
| Add Multiple Choice Modeling to the pretraining baseline (ablation) | MuLTI-B · MSVQ | accuracy: 51.93 -> 53.03 | Helped |
| Add Multiple Choice Modeling to the pretraining baseline (ablation) | MuLTI-B · MSRVTT retrieval | G-Mean: 64.04 -> 64.40 | Helped |
| Add MVM alongside MCM (ablation) | MuLTI-B · MSRQ | accuracy: 46.28 -> 46.11 | Hurt |
| Replace Flatten + Encoder with TGMS (ablation) | MuLTI · MSRQ | accuracy: 44.84 -> 45.54 | Helped |
| Replace Flatten + Encoder with TGMS (ablation) | MuLTI · MSVQ | accuracy: 48.35 -> 49.86 | Helped |
| Add Attention-Adapter to the adapter (ablation) | MuLTI · MSRQ | accuracy: 45.61 -> 45.71 | Helped |
| Add Attention-Adapter to the adapter (ablation) | MuLTI · MSVQ | accuracy: 50.48 -> 50.63 | Helped |
| Freeze 9/12 text encoder layers rather than 6/12 (ablation) | MuLTI with 9/12 frozen video encoder layers · MSRQ | accuracy: 45.54 -> 45.04 | Hurt |
| Add OCR transcripts (ablation) | MuLTI-B · Multi-label classification | mAP: 64.60 -> 67.86 | Helped |
Surprising findings
- Masked frame modeling degraded performance when added to pretraining, so the authors dropped it.
- Unfreezing more encoder layers beyond a point reduced accuracy, attributed to overfitting.
Limitations
- Industrial multi-label dataset is in-house and unreleased, limiting reproducibility.
- Pretraining uses a relatively small set (WebVid-2M and CC-3M), and retrieval results for some baselines are reported with post-processing (DSL, QB-Norm) that differ across methods.
Experimental setup
Datasets: WebVid-2M CC-3M MSRVTT-QA MSVD-QA TGIF-QA MSRVTT DiDeMo in-house short video multi-label dataset
Benchmarks: MSRVTT-QA MSVD-QA TGIF-QA MSRVTT DiDeMo in-house multi-label classification
Metrics: accuracy R@1 R@5 R@10 G-Mean mAP FLOPs FPS memory usage
Baselines: CLIP4CLIP QB-Norm CAMoE TS2-Net ALPRO VIOLET AllInOne Clover Flamingo FrozenBiLM
Builds on: CLIP BERT ALPRO VIOLET Flamingo
Closest papers by meaning
- Frame-Voyager: Learning to Query Frames for Video Large Language Models (ICLR 2025 · similarity 0.87)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.87)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.87)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.86)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.86)
- HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.