Research map / Vision-language and multimodal / Human and embodied modeling
MICap: A Unified Model for Identity-Aware Movie Descriptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MICap, a single-stage encoder-decoder that jointly generates identity-aware movie captions and fills in person ids (FITB), plus the iSPICE identity-sensitive caption metric, evaluated on LSMDC.
Problem
Identity-aware movie captioning requires predicting character ids in captions across a videoset, but existing pipelines are two-stage (caption with someone tags, then fill identities), which propagates errors; existing metrics also fail to capture identity differences.
Approach
A single shared auto-regressive Transformer decoder trained jointly on fill-in-the-blanks (teacher-forced captions with blanks, loss on blank tokens) and full id-aware caption generation, with a multimodal encoder over CLIP, I3D, and ArcFace face features; plus iSPICE, a SPICE variant that filters tuples lacking person ids.
Key results
- On LSMDC validation, MICap (enc-dec) reaches 69.14 FITB class accuracy vs 64.9 for FillIn on test and 65.9 for FillIn on validation, and joint training reaches 70.01.
- Joint training improves captioning CIDEr from 8.01 (caption only) to 9.09 and iSPICE from 0.777 to 0.788.
- Replacing the two-stage FillIn captioner with MICap raises same-id CIDEr from 7.03 to 8.44 and METEOR from 9.41 to 10.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MICap (enc-dec) instead of FillIn | LSMDC FITB · LSMDC validation | Class accuracy: 65.9 -> 69.14 | Helped |
| use MICap instead of the two-stage FillIn method | LSMDC identity-aware captioning · LSMDC validation | CIDEr: 7.77 -> 9.09 | Helped |
| add visual inputs to the text-only model (ablation) | MICap FITB · LSMDC validation | Class accuracy: 64.45 -> 69.14 | Helped |
| use face clusters of [29] (ablation) | MICap FITB · LSMDC validation | Class accuracy: 64.45 -> 67.29 | Helped |
| replace face clusters with raw face detections (ablation) | MICap FITB · LSMDC validation | Class accuracy: 66.56 -> 67.48; 67.29 -> 67.35 | Mixed |
| add CLIP features to the memory (ablation) | MICap FITB · LSMDC validation | Class accuracy: 67.48 -> 67.38; 67.35 -> 67.70 | Mixed |
| replace FaceNet with ArcFace features (ablation) | MICap FITB · LSMDC validation | Class accuracy: 67.38 -> 68.94; 67.70 -> 69.14 | Helped |
| jointly train FITB and full caption generation (ablation) | MICap · LSMDC validation | Class accuracy: 69.14 -> 70.01 | Helped |
| jointly train FITB and full caption generation (ablation) | MICap · LSMDC validation | CIDEr: 8.01 -> 9.09 | Helped |
| jointly train FITB and full caption generation (ablation) | MICap · LSMDC validation | iSPICE: 0.777 -> 0.788 | Helped |
| provide all features instead of only CLIP features (ablation) | T5-Base identity-aware captioning · LSMDC validation | CIDEr: 4.9 -> 4.5 | Hurt |
| provide all features instead of only CLIP features (ablation) | GPT2 identity-aware captioning · LSMDC validation | CIDEr: 3.6 -> 4.4 | Helped |
Surprising findings
- Visual features add only about 1.5% FITB accuracy to FillIn, suggesting BERT-based blank embeddings largely rely on text.
- Simple T5 or GPT-2 adaptations and VLM-style prefix captioners underperform MICap on identity-aware captioning.
Limitations
- Evaluated only on LSMDC validation since test labels and evaluation server are unavailable.
- Captions are limited in vocabulary richness (e.g., 'smiles' instead of 'beams cheerily'), and n-gram and SPICE-style metrics may miss semantic correctness.
- Relies on DBSCAN face clustering across the videoset, whose errors are not fully resolved.
Experimental setup
Datasets: LSMDC
Benchmarks: LSMDC
Metrics: FITB same accuracy FITB diff accuracy FITB instance accuracy FITB class accuracy CIDEr METEOR SPICE iSPICE
Baselines: FillIn FillIn text-only Yu et al. Brown et al. T5-Base GPT-2 CLIPCap-style prefix captioner
Builds on: FillIn (Park et al.) SPICE Transformer BERT
Closest papers by meaning
- IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model (ICLR 2025 · similarity 0.86)
- Progress-Aware Video Frame Captioning (CVPR 2025 · similarity 0.83)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.83)
- Advancing Narrative Long Video Generation via Training-Free Identity-Aware Memory (NeurIPS 2026 · similarity 0.83)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.83)
- Illuminating Visual Identity in Universal Multimodal Embeddings (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.