Research map / Video / Video-language understanding
As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does
Zhongrui Gui, Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a character naming task for films, where models must identify principal characters and assign names from the movie itself without external cast lists or photos.
Problem
Identifying principal characters and assigning their names in films without external cast lists, actor photos, or pre-built character banks, since characters are introduced incrementally as the story unfolds.
Approach
Frames character naming as a task: given a name mentioned in dialogue and the associated video context, the model must determine which visible character bears that name, using only information from the movie itself.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RL (GRPO) stage after SFT (ablation) | Qwen3VL-4B-Thinking · MCN test (character naming) | overall accuracy: 64.16 -> 72.76 | Improved |
| add color-permutation augmentation during SFT (ablation) | Qwen3VL-4B-Thinking SFT · MCN test (character naming) | overall accuracy: 62.39 -> 64.16 | Improved |
| add color-permutation augmentation during RL (ablation) | Qwen3VL-4B-Thinking SFT+RL · MCN test (character naming) | overall accuracy: 71.37 -> 72.76 | Improved |
| use smaller 4B Thinking with SFT+RL instead of naive SFT on 8B Instruct (ablation) | character naming model · MCN test (character naming) | overall accuracy: 64.92 -> 72.76 | Improved |
| naive fine-tuning of Qwen3VL-8B-Instruct on MCN training set | Qwen3VL-8B-Instruct · MCN test (character naming) | overall accuracy: 53.60 -> 64.92 | Improved |
| add face-track filtering before clustering (ablation) | streaming character bank · MCN First Occurrence | mTIoU_n: 52.27 -> 54.40 | Improved |
| add query name filtering (drop off-screen names) (ablation) | streaming character bank · MCN First Occurrence | PCR@10: 49.46 -> 54.54 | Improved |
| add query name filtering (drop off-screen names) (ablation) | streaming character bank · MCN First Occurrence | mTIoU_v: 85.13 -> 84.90 | No clear effect |
| add inter-entry name disambiguation (ablation) | streaming character bank · MCN First Occurrence | mTIoU_n: 52.40 -> 54.40 | Improved |
| add inter-entry name disambiguation (ablation) | streaming character bank · MCN First Occurrence | mTIoU_v: 84.90 -> 84.90 | No clear effect |
| streaming pipeline versus Gemini baselines with full-movie access | character bank construction · MCN First Occurrence | PCR@5: 44.00 -> 63.00 | Improved |
| use fine-tuned MLLM naming instead of subtitle-only MIL naming | character bank construction · MCN First Occurrence | mTIoU_n: 33.38 -> 54.40 | Improved |
Closest papers by meaning
- MICap: A Unified Model for Identity-Aware Movie Descriptions (CVPR 2024 · similarity 0.87)
- Visual Objectification in Films: Towards a New AI Task for Video Interpretation (CVPR 2024 · similarity 0.85)
- Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind (ICML 2024 · similarity 0.85)
- Evaluating Multimodal Narrative Understanding of Popular Hollywood Films (NeurIPS 2026 · similarity 0.84)
- IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model (ICLR 2025 · similarity 0.84)
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
