Research map / Video / Video-language understanding
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a region-level video instruction dataset (VideoRefer-700K) via a multi-agent data engine, a spatial-temporal object encoder Video LLM (VideoRefer), and a benchmark (VideoRefer-Bench) for fine-grained object understanding in videos.
Problem
Video LLMs focus on holistic scene understanding and struggle with fine-grained object-level spatial and temporal referring, and lack high-quality object-level video instruction data and comprehensive benchmarks.
Approach
A multi-agent data engine of off-the-shelf expert models (LLM analyzer, captioner, grounding+SAM segmentor, reviewer, GPT-4o refiner) produces object-level mask-text video data; the model adds a mask-pooling Spatial Token Extractor and a cosine-similarity Temporal Token Merge module to feed object tokens into a Video LLM.
Key results
- VideoRefer-7B reaches 3.42 average on VideoRefer-BenchD single-frame, above GPT-4o (2.95) and InternVL2-26B (2.84).
- VideoRefer-7B achieves 71.9 average accuracy on VideoRefer-BenchQ, versus 71.3 for GPT-4o.
- On VideoMME, VideoRefer reports 55.9/57.6 versus 54.9/56.4 for VideoLLaMA2.1, with gains of +1.4% on Perception-Test and +2.3% on MVBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VideoRefer instead of GPT-4o | Video LLM · VideoRefer-BenchD single-frame | Avg.: 2.95 -> 3.42 | Helped |
| use VideoRefer instead of GPT-4o | Video LLM · VideoRefer-BenchD multi-frame | Avg.: 3.25 -> 3.46 | Helped |
| use VideoRefer instead of GPT-4o | Video LLM · VideoRefer-BenchQ | Average: 71.3 -> 71.9 | Helped |
| use VideoRefer instead of Artemis | video referring model · HC-STVG test | CIDEr: 53.2 -> 68.6 | Helped |
| use VideoRefer instead of VideoLLaMA2.1 | general video understanding model · Perception-Test | score: +1.4% | Helped |
| use VideoRefer instead of VideoLLaMA2.1 | general video understanding model · MVBench | score: +2.3% | Helped |
| use VideoRefer instead of VideoLLaMA2.1 | general video understanding model · VideoMME | score: +1.0%/+1.2% | Helped |
| use multi-frame mode instead of single-frame mode (ablation) | VideoRefer · VideoRefer-BenchD | TD: 3.03 -> 3.10 | Helped |
| use multi-frame mode instead of single-frame mode (ablation) | VideoRefer · VideoRefer-BenchD | HD: 2.97 -> 3.04 | Helped |
| use multi-frame mode instead of single-frame mode (ablation) | VideoRefer · VideoRefer-BenchD | Avg.: 3.42 -> 3.46 | Helped |
| use multi-frame mode instead of single-frame mode (ablation) | VideoRefer · VideoRefer-BenchQ | SQ: 68.3 -> 70.6 | Helped |
| use multi-frame mode instead of single-frame mode (ablation) | VideoRefer · VideoRefer-BenchQ | RQ: 59.1 -> 60.5 | Helped |
Surprising findings
- Video-level referring baseline Artemis performs poorly on general video understanding benchmarks, while VideoRefer improves over its base model.
Limitations
- Reliance on GPT-4o for both refinement of training data and evaluation scoring.
- Object-level QA benchmark is small (198 videos, 1,000 questions) and description benchmark has 400 entries.
Experimental setup
Datasets: Panda-70M MeViS Ref-YouTube-VOS A2D-Sentence DAVIS-2017 HC-STVG Perception-Test MVBench VideoMME
Benchmarks: VideoRefer-BenchD VideoRefer-BenchQ HC-STVG Perception-Test MVBench VideoMME
Metrics: GPT-4o 0-5 scores (SC, AD, TD, HD) multiple-choice accuracy BLEU@4 METEOR ROUGE-L CIDEr SPICE
Baselines: GPT-4o GPT-4o-mini InternVL2-26B Qwen2-VL-7B LLaVA-OV-7B LongVA-7B LongVU-7B Osprey-7B Ferret-7B Elysium-7B Artemis-7B Merlin
Builds on: VideoLLaMA2.1 Osprey SAM 2 Panda-70M
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.89)
- Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025 · similarity 0.89)
- VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation (CVPR 2025 · similarity 0.89)
- DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences (NeurIPS 2026 · similarity 0.89)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.