Research map / Vision-language and multimodal / Human and embodied modeling
AutoAD III: The Prequel - Back to the Pixels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds two pixel-based movie audio description datasets (CMD-AD via audio-audio alignment, HowTo-AD via rewriting HowTo100M captions), trains Q-former-based Movie-BLIP2 and Movie-Llama2 models with character banks, and proposes CRITIC and LLM-AD-eval metrics.
Problem
Automatic audio description (AD) generation for movies needs fine-grained visual grounding and character naming, but suitable pixel-level training data is scarce and existing evaluation metrics (n-gram captioning scores, retrieval) do not capture AD quality.
Approach
Align existing audio-only AD tracks with public movie clips via a two-stage text-then-audio (WER sliding window + RANSAC on mel-spectrogram correlation) pipeline, convert HowTo100M captions into character-named pseudo-AD, and train a character-aware Q-former VLM on raw frames plus face exemplars, with a co-reference-based character naming metric (CRITIC).
Key results
- Movie-Llama2 reaches CRITIC 32.7 and CIDEr 25.0 on CMD-AD-Eval after HowTo-AD pretraining, versus 8.2 CRITIC and 13.5 CIDEr for AutoAD-II.
- HowTo-AD pretraining raises Movie-Llama2 CRITIC from 25.2 to 32.7 on CMD-AD-Eval.
- On MAD-Eval, Movie-Llama2 achieves CIDEr 24.0 and R@5/16 52.8, outperforming AutoAD-II (19.2 CIDEr, 51.3 R@5/16) without MAD training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AutoAD-II with Movie-BLIP2 | Movie AD generation · CMD-AD-Eval | CIDEr: 13.5 -> 22.3 | Helped |
| Replace AutoAD-II with Movie-Llama2 | Movie AD generation · CMD-AD-Eval | CIDEr: 13.5 -> 25.0 | Helped |
| Use Movie-Llama2 instead of AutoAD-II | Movie AD generation · MAD-Eval | CIDEr: 19.2* -> 24.0* | Helped |
| Pretrain Movie-BLIP2 on HowTo-AD (ablation) | Movie-BLIP2 · CMD-AD-Eval | CRITIC: 24.5 -> 30.2 | Helped |
| Pretrain Movie-BLIP2 on HowTo-AD (ablation) | Movie-BLIP2 · CMD-AD-Eval | CIDEr: 21.2 -> 22.3 | Helped |
| Pretrain Movie-Llama2 on HowTo-AD (ablation) | Movie-Llama2 · CMD-AD-Eval | CRITIC: 25.2 -> 32.7 | Helped |
| Pretrain Movie-Llama2 on HowTo-AD (ablation) | Movie-Llama2 · CMD-AD-Eval | CIDEr: 21.7 -> 25.0 | Helped |
| Pretrain Movie-Llama2 on HowToCaption without character banks or rewritten captions (ablation) | Movie-Llama2 · CMD-AD-Eval | CIDEr: 21.7 -> 20.8 | Hurt |
| Pretrain Movie-Llama2 on HowToCaption without character banks or rewritten captions (ablation) | Movie-Llama2 · CMD-AD-Eval | CRITIC: 25.2 -> 25.6 | No clear effect |
Surprising findings
- Pretraining on the HowToCaption subset without character banks or caption rewriting gives little benefit on movie AD (CRITIC 25.6 vs 32.7 with HowTo-AD).
Limitations
- Audio-audio alignment succeeds for only about 80% of overlapping movies.
- AD coherence across the movie is not modeled, so narrations may repeat information.
- Only projection heads are trained; visual backbone and LLM are frozen.
Experimental setup
Datasets: CMD-AD HowTo-AD HowTo100M CMD AudioVault-8K MAD LSMDC
Benchmarks: CMD-AD-Eval MAD-Eval
Metrics: CRITIC LLM-AD-eval CIDEr Recall@k/N
Baselines: AutoAD-II AutoAD-I MM-Narrator-GPT4 Video-BLIP2 Video-Llama2
Builds on: BLIP-2 MiniGPT-4 AutoAD-II VideoLLaMA HowToCaption
Closest papers by meaning
- DistinctAD: Distinctive Audio Description Generation in Contexts (CVPR 2025 · similarity 0.89)
- Contextual AD Narration with Interleaved Multimodal Sequence (CVPR 2025 · similarity 0.89)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.86)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.86)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.