Research map / Video / Video-language understanding
Contextual AD Narration with Interleaved Multimodal Sequence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Uni-AD, which feeds interleaved multimodal sequences (video tokens, character portraits, text, and past context) into a frozen LLM to generate character-aware audio descriptions, with a character-refinement module and contrastive loss.
Problem
Automatic audio description generation for long-form movies needs character naming, contextual coherence with the storyline, and avoidance of repetitive narration, which prior adapter-based, Q-former, and training-free GPT-4 approaches handle poorly.
Approach
Formulates all inputs (current video clip, character exemplar images with names, past video and ADs) as one interleaved multimodal prompt for a frozen LLM, uses a lightweight transformer visual mapping network with learnable latent vectors, filters characters with a cross-attention refinement module, and adds a contrastive loss to discourage repeated ADs.
Key results
- On MAD-eval-Named with context, Uni-AD (LLaMA2 & CLIP-L14) reaches 17.1 ROUGE-L, 28.2 CIDEr and 54.2 R@5/16, exceeding AutoAD-II (13.4/19.5/50.8) and MM-Narrator (13.4/13.9/49.0).
- On CMDAD, Uni-AD obtains CRITIC 41.9 and LLM-AD-eval 2.92, competitive with AutoAD-Zero (43.7 CRITIC) and AutoAD-III.
- Character-refinement module improves GPT-2-B32 Uni-AD CIDEr from 18.5 to 23.7 without context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Uni-AD instead of AutoAD-II | GPT-2 & CLIP-B32 on MAD-eval-Named · MAD-eval-Named | CIDEr: 19.2 -> 24.0 | Helped |
| Use Uni-AD instead of AutoAD-III | MAD-eval-Named · MAD-eval-Named | CIDEr: 24.0 -> 24.0 | No clear effect |
| Use contextual Uni-AD instead of MM-Narrator | MAD-eval-Named · MAD-eval-Named (context) | CIDEr: 13.9 -> 28.2 | Helped |
| Replace AutoAD-III with Uni-AD | CMDAD · CMDAD | CRITIC: 25.2 -> 41.9 | Mixed |
| Replace AutoAD-III with Uni-AD | TVAD · TVAD | CIDEr: 26.1 -> 26.6 | Helped |
| Add character-refinement module (ablation) | Uni-AD with GPT-2 & CLIP-B32 on MAD-eval-Named · MAD-eval-Named | CIDEr: 18.5 -> 23.7 | Helped |
| Add character-refinement module (ablation) | Uni-AD with LLaMA2 & CLIP-L14 on MAD-eval-Named · MAD-eval-Named | CIDEr: 22.6 -> 25.9 | Helped |
| Increase visual mapping latents from 1 to 30 (ablation) | Uni-AD with GPT-2 & CLIP-B32 on MAD-eval-Named · MAD-eval-Named | CIDEr: 19.0 -> 23.7 | Helped |
| Increase visual mapping latents from 1 to 60 (ablation) | Uni-AD with LLaMA & CLIP-L14 on MAD-eval-Named · MAD-eval-Named | R@5/16: 51.5 -> 53.7 | Helped |
| Add 1 context video (ablation) | Uni-AD without contrastive loss on MAD-eval-Named · MAD-eval-Named | CIDEr: 27.3 -> 27.5 | Helped |
| Add contrastive loss (ablation) | Uni-AD with 1 context video on MAD-eval-Named · MAD-eval-Named | CIDEr: 27.5 -> 28.2 | Helped |
| Use 3 recurrent context ADs instead of 1 (ablation) | Uni-AD on MAD-eval-Named · MAD-eval-Named | CIDEr: 25.8 -> 23.1 | Hurt |
Surprising findings
- Recurrent use of predicted past ADs as context decreases performance, attributed to train-test discrepancy.
- Increasing the number of visual latent vectors helps Uni-AD but leaves AutoAD-II performance essentially unchanged.
Limitations
- Character-refinement performance is modest (0.41 precision, 0.77 recall), leaving a large gap to ground-truth character information.
- Recurrent context using predicted past ADs hurts performance due to train-test discrepancy.
Experimental setup
Datasets: AudioVault-AD MAD-v2-Named MAD-eval-Named CMDAD TVAD MovieNet
Benchmarks: MAD-eval-Named CMDAD TVAD
Metrics: ROUGE-L CIDEr R@k/N CRITIC LLM-AD-eval
Baselines: ClipCap AutoAD-I AutoAD-II AutoAD-III AutoAD-Zero MM-Narrator
Builds on: AutoAD-II ClipCap Flamingo
Closest papers by meaning
- DistinctAD: Distinctive Audio Description Generation in Contexts (CVPR 2025 · similarity 0.90)
- AutoAD III: The Prequel - Back to the Pixels (CVPR 2024 · similarity 0.89)
- MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning (CVPR 2024 · similarity 0.86)
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.85)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.83)
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.