Research map / Vision-language and multimodal / Multimodal LLMs

Evaluating Multimodal Narrative Understanding of Popular Hollywood Films

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a public-domain, box-office-popular Hollywood film collection (61 films) from Variety box office data and a 628-question multimodal MCQ benchmark on classical Hollywood narrative, showing top models reach 61.1% vs 82% human.

Problem

Multimodal benchmarks built on Hollywood films are unstable due to copyright and DMCA takedowns, and existing benchmarks lack focus on narrative elements relevant to film scholarship.

Approach

Select films by historical box office popularity (extracted from Variety with multimodal LLMs) and verify likely public domain status via Copyright Catalog renewal records, then build narrative-focused MCQs over them and filter out questions answerable from parametric memory.

Key results

• Gemini 3 Flash on full video with audio reaches 61.1% accuracy, the best overall, versus 82.0% human performance.
• Many long-video and open VLMs perform near chance (roughly 21-29%) in the closed-book and end-to-end settings.
• Clip-based Socratic captions with Gemini 3 Flash reach 58.9%, statistically comparable to native full-video processing.

Surprising findings

• Frame-image retrieval via VideoAgent barely exceeds chance across all backbones, since CLIP similarity cannot capture audio-dependent content.
• Closed-book accuracy on the filtered benchmark is near chance for all models, yet per-film closed-book accuracy correlates across backbones, suggesting shared parametric film knowledge.

Limitations

• Popularity-based selection excludes non-mainstream and race films and encodes only one form of cultural significance.
• Questions were written by researchers at US universities, shaping which narrative aspects are salient.
• Benchmark covers only Classical Hollywood era (through 1963), limiting generalization to other periods.

Experimental setup

Datasets: Variety box office scans (1922-1979) Catalog of Copyright Entries Copyright Public Records System AFI catalog

Benchmarks: Classical Hollywood Narrative Benchmark (CHNB)

Metrics: accuracy Spearman rank correlation Movie F1 Tuple F1

Baselines: HourLLaVA VAMBA-Qwen2-VL-7B VideoChat-Flash LLaVA-NeXT-Video-DPO Qwen3-VL-8B GLM-4.1V-9B-Thinking GPT-5-mini Claude Haiku 4.5 Gemini 3 Flash

Builds on: Socratic models VideoAgent Letta

Closest papers by meaning

• CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (NeurIPS 2025 · similarity 0.86)
• ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models (NeurIPS 2025 · similarity 0.85)
• Visual Objectification in Films: Towards a New AI Task for Video Interpretation (CVPR 2024 · similarity 0.84)
• MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences (ICLR 2025 · similarity 0.83)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.83)
• ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026 · similarity 0.83)