Research map / Vision-language and multimodal / Multimodal LLMs
Mind the Gap: Dataset and Fine-grained Evaluation for Inline Audio Descriptions
Subhashini Venugopalan, Yingwen Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a dataset and fine-grained evaluation framework for inline audio descriptions (ADs that must fit within silences) in diverse user-generated video content, targeting MLLM performance.
Problem
MLLM-generated audio descriptions for blind and low-vision users must fit inside existing silences in videos (inline ADs), but their performance on this setting, especially for diverse user-generated content, is under-explored.
Approach
A dataset of inline audio descriptions for user-generated content paired with a fine-grained evaluation protocol for assessing MLLM-generated inline ADs; specific method details are not available from the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add expert AD guidelines to the prompt (ablation) | Gemini-2.5 Flash · MTG-AD validation | Audio Overlaps (s): 15.07 -> 59.19 | Hurt |
| add expert AD guidelines to the prompt (ablation) | Gemini-2.5 Flash · MTG-AD validation | On-Screen Text: 0.47 -> 0.69 | Improved |
| add timestamped ASR transcript and speaking-rate target (ablation) | Gemini-2.5 Flash · MTG-AD validation | Audio Overlaps (s): 59.19 -> 21.34 | Improved |
| add explicit per-gap duration and word budget steps (ablation) | Gemini-2.5 Flash · MTG-AD validation | Audio Overlaps (s): 21.34 -> 12.77 | Improved |
| add explicit per-gap word budget on top of Guideline + ASR (ablation) | Gemini-2.5 Flash · MTG-AD validation | Avg. Silence (s): 6.71 -> 8.97 | Hurt |
| use next-generation omni-modal model (Gemini-3.5 Flash) | Gemini-2.5 Flash · MTG-AD test | Audio Overlaps (s): 8.92 -> 0.26 | Improved |
| use Qwen-3.5-Omni-Flash instead of Gemini-2.5 Flash | MLLM AD generator · MTG-AD test | Audio Overlaps (s): 8.92 -> 52.14 | Hurt |
| use reference-free Quality-RF instead of reference-based VideoA11y-Ref Accurate | LLM judge · BLV perceived accuracy | Spearman rho: -0.264 -> +0.244 | Improved |
| use Quality-RF instead of VideoA11y-Ref for Descriptive | LLM judge · BLV descriptive rating | Spearman rho: +0.340 -> +0.583 | Improved |
| replace human AD with AutoAD (Gemini-2.5 Flash, Gd. + Timing ASR) | audio description source · BLV study | preference: 53.6% preferred human | Mixed |
Closest papers by meaning
- AutoAD III: The Prequel - Back to the Pixels (CVPR 2024 · similarity 0.86)
- Contextual AD Narration with Interleaved Multimodal Sequence (CVPR 2025 · similarity 0.86)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.86)
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.86)
- DistinctAD: Distinctive Audio Description Generation in Contexts (CVPR 2025 · similarity 0.86)
- AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
