Research map / Video / Video-language understanding
MAMS: Model-Agnostic Module Selection Framework for Video Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAMS, a model-agnostic framework that adaptively selects between large and small caption generation modules and token subsets per video, plus an adaptive attention mask, improving SwinBERT, UniVL, and mPLUG-2 on MSVD, MSRVTT, and YouCookII.
Problem
Video captioning models use a fixed number of frames for all videos, so sparse sampling can miss key frames and dense sampling yields redundant visual tokens, causing performance saturation or degradation.
Approach
Augment an existing end-to-end captioner with a smaller parallel generation module and Gumbel-Softmax-based module and token selectors that choose frame subsets per video, trained by routing loss to only one module, together with an adaptive attention mask built from token significance scores (attention from CLS times visual token norm).
Key results
- MAMS + mPLUG-2 reaches CIDEr 171.6 on MSVD (vs 165.8 for mPLUG-2) and 82.9 on MSRVTT (vs 80.3).
- MAMS + SwinBERT improves CIDEr on YouCookII from 109.0 to 116.7 and on MSVD from 120.6 to 125.0.
- Swapping the module selector conditions drops MAMS+SwinBERT CIDEr on MSVD from 125.0 to 119.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAMS to SwinBERT | SwinBERT · MSVD | BLEU-4: +2.7 | Helped |
| Add MAMS to mPLUG-2 | mPLUG-2 · MSRVTT | METEOR: (-0.2) | Hurt |
| Add MAMS to SwinBERT | SwinBERT · MSRVTT | METEOR: (-0.1) | Hurt |
| Add MAMS to SwinBERT | SwinBERT · YOUCOOKII | BLEU-4: +3.5 | Helped |
| Add MAMS to UniVL | UniVL · YOUCOOKII | BLEU-4: +3.2 | Helped |
| Add the adaptive attention mask to MAMS + SwinBERT (ablation) | MAMS + SwinBERT · MSVD | METEOR: 41.6 -> 42.1 | Helped |
| Add the adaptive attention mask to standalone SwinBERT (ablation) | SwinBERT · MSVD | BLEU-4: 58.2 -> 61.1 | Helped |
| Swap the module selector conditions (ablation) | MAMS + SwinBERT · MSVD | BLEU-4: 60.9 -> 58.1 | Hurt |
| Invert the token significance score (ablation) | MAMS + SwinBERT · MSVD | BLEU-4: 60.9 -> 55.8 | Hurt |
| Use three candidate generation modules instead of the default two (ablation) | MAMS + SwinBERT · MSVD | BLEU-4: 60.9 -> 59.1 | Hurt |
| Use four candidate generation modules instead of the default two (ablation) | MAMS + SwinBERT · MSVD | BLEU-4: 60.9 -> 57.4 | Hurt |
Surprising findings
- Performance of existing captioners stagnates or declines as the number of frames increases, motivating adaptive frame counts.
- Increasing the number of candidate generation modules to three or four degrades performance relative to the two-module default.
Limitations
- Gains on MSRVTT are modest for METEOR and ROUGE, and METEOR slightly decreases in some rows.
- Requires training two generation modules and the selector, adding training complexity; the number of candidate modules is limited by data.
Experimental setup
Datasets: MSVD MSRVTT YouCookII
Benchmarks: MSVD MSRVTT YouCookII
Metrics: BLEU-4 METEOR ROUGE CIDEr
Baselines: SwinBERT UniVL mPLUG-2 TextKG CoCap VIOLETv2 EMCL-Net CLIP-DCD
Builds on: SwinBERT UniVL mPLUG-2
Closest papers by meaning
- M-LLM Based Video Frame Selection for Efficient Video Understanding (CVPR 2025 · similarity 0.87)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.86)
- Adaptive Keyframe Sampling for Long Video Understanding (CVPR 2025 · similarity 0.86)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.85)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.85)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.