Research map / Video / Video-language understanding
Artemis: Towards Referential Understanding in Complex Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Artemis is an MLLM for video-based referential understanding that tracks a box-specified target across the video, selects diverse RoIs via K-means, and is trained in three stages on the new VideoRef45K dataset.
Problem
Video-based referential understanding: describing the action of a target specified by a bounding box in one frame across a complex, multi-action video, which existing image-based or multi-frame MLLMs handle poorly.
Approach
Extract compact target-specific spatiotemporal features by tracking the referred box through the video with an off-the-shelf tracker (HQTrack), then K-means-selecting a diverse subset of RoIs and feeding them via RoIAlign as tokens alongside spatial and temporal video features into a Vicuna-7B MLLM trained in three stages.
Key results
- On HC-STVG, Artemis reaches BERTScore 0.9135, BLEU@4 15.5, CIDEr 53.2, outperforming Merlin (0.8829, 3.3, 10.5) and image-based Osprey, Ferret, and Shikra on all metrics.
- K-means RoI selection with 4 RoIs gives the best ablation results on HC-STVG (BLEU@4 14.6 and CIDEr 47.2 with clustering, versus 13.9 and 43.7 without RoI selection).
- On MSVD-QA, Artemis-style VideoRoI reports 72.1 accuracy versus 64.9 for Video-ChatGPT, zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train for video-based referring with Artemis | Artemis · HC-STVG test | BLEU@4: 3.3 -> 15.5 | Helped |
| use K-means RoI selection instead of no selection (ablation) | Artemis · HC-STVG | BLEU@4: 13.9 -> 14.6 | Helped |
| use K-means RoI selection instead of uniform sampling (ablation) | Artemis · HC-STVG | BLEU@4: 14.2 -> 14.6 | Helped |
| use K-means RoI selection instead of random sampling (ablation) | Artemis · HC-STVG | BLEU@4: 14.3 -> 14.6 | Helped |
| use 2 rather than 4 selected RoIs (ablation) | Artemis · HC-STVG | BLEU@4: 14.3 -> 14.6 | Helped |
| use 1 rather than 4 selected RoIs (ablation) | Artemis · HC-STVG | BLEU@4: 14.3 -> 13.9 | Hurt |
| use 6 rather than 4 selected RoIs (ablation) | Artemis · HC-STVG | BLEU@4: 14.3 -> 14.3 | No clear effect |
| use 8 rather than 4 selected RoIs (ablation) | Artemis · HC-STVG | BLEU@4: 14.3 -> 14.0 | Hurt |
Surprising findings
- Adding image-based referring outputs integrated by GPT-3.5-Turbo fails to capture video-level logic and can hallucinate events.
- Using more RoIs (6 or 8) gives slightly worse metrics than 4, because redundant information is introduced.
Limitations
- Relies on an off-the-shelf tracker whose inaccurate boxes can mislead the model.
- Suffers from spatial-temporal aliasing, e.g., reversing temporal order of actions.
- Comparisons to image-based referring models are not fully fair since those models were not trained for video referring.
Experimental setup
Datasets: HC-STVG VID-Sentence A2D Sentences LaSOT MeViS GOT-10K MGIT LAION-CCSBU Video-LLaVA video-text pairs LLaVA-1.5 instruction data Video-ChatGPT instruction set MSVD-QA
Benchmarks: HC-STVG Video-ChatGPT benchmark MSVD-QA MSRVTT-QA ActivityNet-QA
Metrics: BERTScore BLEU@4 METEOR ROUGE_L CIDEr SPICE correctness detail orientation contextual understanding temporal understanding
Baselines: Osprey Ferret-13B Shikra-7B Merlin Video-Chat LLaMA-Adapter Video-LLaMA Video-ChatGPT Valley-v3 FrozenBiLM
Builds on: Video-ChatGPT Video-LLaVA LLaVA-1.5 HQTrack Merlin
Closest papers by meaning
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.87)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.86)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.