Research map / Vision-language and multimodal / Human and embodied modeling
DistinctAD: Distinctive Audio Description Generation in Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DistinctAD, a two-stage framework that adapts CLIP's vision encoder to movie audio-description data and generates context-distinctive ADs for consecutive clips via a Contextual EMA module and a distinctive word prediction loss.
Problem
Automatic audio description generation for long movies suffers from a domain gap between movie-AD data and VLM training data, and from repetitive descriptions across highly similar neighboring visual clips.
Approach
Adapt the CLIP vision encoder to movie-AD data with global video-AD and fine-grained frame-word matching (no extra AD corpora), then apply Expectation-Maximization Attention across N consecutive clips to extract common bases that reduce redundancy, plus an auxiliary loss that explicitly predicts words unique to the current AD.
Key results
- On MAD-Eval, DistinctAD with CLIPAD-B16 and LLaMA3-8B reaches CIDEr 27.3 and R@5/16 56.0, state of the art among compared methods.
- With the same CLIP-B32 and GPT-2 backbone, DistinctAD gets CIDEr 24.5 vs AutoAD-II 19.5; Stage-I CLIP-AD adaptation raises it to 25.5.
- On CMD-AD and TV-AD, DistinctAD (LLaMA3) achieves CIDEr 22.7 and 27.4 and R@1/5 of 33.0 and 32.1, outperforming AutoAD-Zero and AutoAD-III on R@1/5 and CIDEr except AutoAD-III pre-trained on 3.4M HowTo-AD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt the CLIP vision encoder to movie-AD data (ablation) | DistinctAD with CLIP-B32 and GPT-2 · MAD-Eval | CIDEr: 24.5 vs. 25.5 | Helped |
| add global video-AD matching loss (ablation) | Stage-I CLIP-AD adaptation · MAD-Eval-Named | CIDEr: 6.7 to 8.2 | Helped |
| add fine-grained frame-AD matching loss (ablation) | Stage-I CLIP-AD adaptation · MAD-Eval-Named | CIDEr: 6.7 to 7.7 | Helped |
| combine global and fine-grained matching losses (ablation) | Stage-I CLIP-AD adaptation · MAD-Eval-Named | CIDEr: 6.7 to 8.6 | Helped |
| set matching-loss coefficient γ to 0.5 rather than 0.1, 0.3, or 0.7 (ablation) | Stage-I CLIP-AD adaptation · MAD-Eval-Named | CIDEr: 0.1: 8.0; 0.3: 8.5; 0.5: 8.6; 0.7: 7.7 | Mixed |
| add reconstructed compact features (ablation) | Stage-II DistinctAD · MAD-Eval-Named | CIDEr: 23.1 to 23.7 | Helped |
| add cross-attention features over bases (ablation) | Stage-II DistinctAD without distinctive word loss · MAD-Eval-Named | CIDEr: 23.1 to 23.4 | Helped |
| add distinctive word prediction loss (ablation) | Stage-II DistinctAD · MAD-Eval-Named | R@5/16: 46.2 to 49.5 | Helped |
| combine cross-attention features with distinctive word prediction loss (ablation) | Stage-II DistinctAD · MAD-Eval-Named | CIDEr: 23.4 to 25.3 | Helped |
| use the full Stage-II pipeline (ablation) | DistinctAD · MAD-Eval-Named | R@5/16: 46.2 to 51.7 | Helped |
| sample non-consecutive rather than consecutive context clips (ablation) | DistinctAD with Contextual EMA · MAD-Eval-Named | CIDEr: 25.5 vs. 23.8 | Hurt |
| sample non-consecutive rather than consecutive context clips (ablation) | DistinctAD with Contextual EMA · MAD-Eval-Named | R@5/16: 51.7 vs. 52.5 | Helped |
Surprising findings
- Using non-consecutive clips lowers CIDEr by 1.7 but raises R@5/16 by 0.8, suggesting diverse context aids distinctiveness at the cost of n-gram overlap.
Limitations
- Requires a large number of parameters, as stated in the conclusion.
- Generated ADs still fall short of human-annotated quality.
- Relies on external character inference results from AutoAD-Zero for face/character prompts.
Experimental setup
Datasets: MAD-v2-Named MAD-Eval-Named CMD-AD TV-AD
Benchmarks: MAD-Eval CMD-AD TV-AD
Metrics: ROUGE-L CIDEr SPICE Recall@k/N (BertScore) LLM-AD-eval
Baselines: MM-Vid MM-Narrator LLM-AD AutoAD-Zero AutoAD-I AutoAD-II AutoAD-III MovieSeq Video-BLIP2 Video-LLaMA2
Builds on: AutoAD Expectation-Maximization Attention CLIP Perceiver
Closest papers by meaning
- Contextual AD Narration with Interleaved Multimodal Sequence (CVPR 2025 · similarity 0.90)
- AutoAD III: The Prequel - Back to the Pixels (CVPR 2024 · similarity 0.89)
- MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning (CVPR 2024 · similarity 0.86)
- Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions (CVPR 2025 · similarity 0.84)
- Looking Similar Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning (CVPR 2024 · similarity 0.84)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.