Research map / Vision-language and multimodal / Multimodal LLMs
IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ID-aware visual instruction tuning (IDA-VLM, built on Qwen-VL-Chat with an added ID-Former) and the MM-ID benchmark to test identity memory and recognition across matching, location, QA, and captioning.
Problem
Current LVLMs process a single scenario and cannot associate and recognize the same character or instance across different images given ID reference images, which is needed for movie understanding with multiple characters.
Approach
Visual instruction tuning with ID reference: ID images (name + cropped reference) are provided interleaved with test images; a two-stage data pipeline (rule-based crops from VCR/RefCOCO/Flickr30k, then GPT-4V-annotated MovieNet data) and an ID-Former with a second cross-attention that lets ID-image queries modulate test-image features.
Key results
- IDA-VLM achieves 0.716 matching accuracy, 0.821 location score, 5.71 Q&A and 5.15 caption GPT-4 scores on MM-ID, the best among compared models.
- Gemini-pro reaches 0.687 matching, GPT-4V 0.627 matching and 4.67 caption; QwenVL-Chat 0.504 location, far below IDA-VLM's 0.821.
- Removing the second (MovieNet/GPT-4V) tuning stage drops Q&A from 5.80 to 4.08 and caption from 5.19 to 3.86 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train IDA-VLM with the proposed ID-reference tuning and ID-Former | QwenVL-Chat · MM-ID Matching | accuracy: 0.687 -> 0.716 | Helped |
| Train IDA-VLM with the proposed ID-reference tuning and ID-Former | QwenVL-Chat · MM-ID Location | IoU accuracy: 0.504 -> 0.821 | Helped |
| Train IDA-VLM with the proposed ID-reference tuning and ID-Former | QwenVL-Chat · MM-ID Q&A | GPT-4 absolute score: 4.97 -> 5.71 | Helped |
| Train IDA-VLM with the proposed ID-reference tuning and ID-Former | QwenVL-Chat · MM-ID Caption | GPT-4 absolute score: 4.67 -> 5.15 | Helped |
| Add first-stage tuning (ablation) | IDA-VLM · MM-ID Matching | accuracy: 0.746 -> 0.716 | Hurt |
| Add first-stage tuning (ablation) | IDA-VLM · MM-ID Location | IoU accuracy: 0.813 -> 0.821 | Helped |
| Add first-stage tuning (ablation) | IDA-VLM · MM-ID Q&A (one-test-image subset) | GPT-4 absolute score: 5.43 -> 5.80 | Helped |
| Add first-stage tuning (ablation) | IDA-VLM · MM-ID Caption (one-test-image subset) | GPT-4 absolute score: 5.08 -> 5.19 | Helped |
| Add second-stage tuning (ablation) | IDA-VLM · MM-ID Location | IoU accuracy: 0.715 -> 0.821 | Helped |
| Add second-stage tuning (ablation) | IDA-VLM · MM-ID Q&A (one-test-image subset) | GPT-4 absolute score: 4.08 -> 5.80 | Helped |
| Add second-stage tuning (ablation) | IDA-VLM · MM-ID Caption (one-test-image subset) | GPT-4 absolute score: 3.86 -> 5.19 | Helped |
| Use ID-Former instead of a standard query former (ablation) | IDA-VLM · MM-ID Location | IoU accuracy: 0.803 -> 0.821 | Helped |
Surprising findings
- Removing the first tuning stage slightly improves matching (0.746 vs 0.716), which the authors attribute to first-stage data not targeting matching.
- Gemini-pro scores only 0.081 on location, and GPT-4V only 0.244, despite strong QA performance.
Limitations
- Input limited to eight images total (ID plus test images), so keyframes are used rather than full movie segments.
- Possible slight loss of original Qwen-VL-Chat capabilities after ID tuning.
- MM-ID is small (585 samples) and QA/caption scoring relies on GPT-4 judging.
Experimental setup
Datasets: VCR RefCOCO Flickr30k MovieNet LLaVA instruction data Re-ID datasets animation images from the internet
Benchmarks: MM-ID
Metrics: accuracy IoU (threshold 0.5) GPT-4 absolute score (1-10) GPT-4 relative score ratio
Baselines: MMICL SEED QwenVL-Chat InternLM-XComposer2 QwenVL-Plus QwenVL-Max Gemini-pro GPT-4V
Builds on: Qwen-VL-Chat LLaVA instruction tuning BLIP-2 Q-Former
Closest papers by meaning
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.87)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.87)
- Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge (CVPR 2025 · similarity 0.86)
- Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts (AAAI 2025 · similarity 0.86)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.86)
- Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.