Research map / Video / Video-language understanding
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free MM-Narrator prompts GPT-4 with multimodal in-context learning, short-term text memory and long-term visual register-and-recall memory to generate character-centric audio descriptions for long videos, plus a GPT-4 segment-based evaluator (SegEval).
Problem
Automatic audio description (AD) generation for long-form videos requires story-coherent, character-centric narration over hours of content, which fine-tuned short-clip models handle poorly and which lacks suitable evaluation for recurrent text generation.
Approach
A training-free recurrent narrator that combines a short-term textual memory queue with a long-term visual memory (CLIP-based register-and-recall for character re-identification), and selects few-shot multimodal demonstrations by reasoning-step complexity rather than similarity.
Key results
- MM-Narrator with MM-ICL reaches ROUGE-L 12.1, CIDEr 11.6, SPICE 4.5 on MAD-eval-Named, beating fine-tuned AutoAD-I (R-L 11.9, C 14.3, S 4.4) on R-L and SPICE, though CIDEr is lower.
- Versus training-free LLM/LMM baselines, MM-Narrator with MM-ICL improves CIDEr from 6.1 (MM-Vid, GPT-4V) to 11.6 and R@5/16 to 48.0.
- With an external character bank, MM-Narrator reaches R-L 13.4 and CIDEr 13.9, comparable to AutoAD-II (R-L 13.4, C 19.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MM-ICL to MM-Narrator (ablation) | MM-Narrator · MAD-eval-Named | ROUGE-L: 10.3 -> 12.1 | Helped |
| Add MM-ICL to MM-Narrator (ablation) | MM-Narrator · MAD-eval-Named | CIDEr: 4.9 -> 11.6 | Helped |
| Add MM-ICL to MM-Narrator (ablation) | MM-Narrator · MAD-eval-Named | R@5/16: 47.1 -> 48.0 | Helped |
| Use random demonstrations with CoT rather than without CoT (ablation) | MM-Narrator random MM-ICL · MAD-eval-Named | CIDEr: 12.9±0.2 -> 13.4±0.2 | Helped |
| Use complexity-selected shortest demonstrations with CoT rather than (Q, A) examples (ablation) | MM-Narrator complexity-based MM-ICL · MAD-eval-Named | CIDEr: 13.3±0.3 -> 13.9±0.1 | Helped |
| Select shortest rather than longest demonstrations (ablation) | MM-Narrator complexity-based MM-ICL · MAD-eval-Named | CIDEr: 12.7±0.2 -> 13.9±0.1 | Helped |
| Add an external character bank (ablation) | MM-Narrator · MAD-eval-Named | CIDEr: 11.6 -> 13.9 | Helped |
| Add multiple frames, subtitles, task-specific hint, short-term memory, MM-ICL, and long-term visual memory in the staged build (ablation) | MM-Narrator · MAD-eval-Named | captioning metrics | Mixed |
Surprising findings
- Similarity-based demonstration retrieval did not improve AD generation over random sampling, while complexity-based selection (shortest reasoning chains) did.
- Swapping GPT-4 for GPT-4V as the vision expert lowered classic captioning scores (e.g., CIDEr 11.6 to 7.0) because generated ADs were longer than human references.
Limitations
- Relies on GPT-4 API and multiple off-the-shelf vision/audio experts, making it costly and dependent on external models.
- Classic captioning metrics against short human ADs may penalize more detailed generated descriptions.
- Character identities remain unknown until named in dialogue unless an external character bank is supplied.
Experimental setup
Datasets: MAD-v2-Named MAD-eval-Named
Benchmarks: MAD-eval-Named
Metrics: ROUGE-L CIDEr SPICE METEOR BLEU-1 R@k/N SegEval scores
Baselines: ClipCap ClipDec AutoAD-I AutoAD-II VLog VideoChat MM-Vid GPT-4V
Builds on: AutoAD chain-of-thought k-NN demonstration retrieval CLIP
Closest papers by meaning
- JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation (NeurIPS 2025 · similarity 0.87)
- Contextual AD Narration with Interleaved Multimodal Sequence (CVPR 2025 · similarity 0.86)
- DistinctAD: Distinctive Audio Description Generation in Contexts (CVPR 2025 · similarity 0.86)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.85)
- ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing (NeurIPS 2025 · similarity 0.85)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.