Research map / Video / Video-language understanding
VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes long-duration and high-resolution video instruction data (VISTA-400K) by spatiotemporally combining existing videos and captions via seven augmentation schemes, and fine-tunes three video LMMs, also releasing the HRVideoBench benchmark.
Problem
Video LMMs struggle with long-duration and high-resolution videos, largely due to a lack of high-quality, license-friendly long/high-res video instruction data.
Approach
Create synthetic long and high-resolution videos by temporally concatenating clips and spatially/temporally overlaying needle videos into haystack videos (or tiling 64 clips in an 8x8 grid), then use Gemini-1.5-Pro to write QA pairs from existing captions via text only, yielding a scalable augmentation-based dataset.
Key results
- Finetuning on VISTA-400K gives an average gain of 3.3% across long-video benchmarks (e.g., VISTA-LongVA reaches 55.5 on Video-MME w/o subtitles vs 52.4 baseline).
- On the new HRVideoBench, finetuned models gain 6.5% on average; VISTA-VideoLLaVA improves from 32.5 to 47.5.
- Ablation on Mantis-Idefics2 shows removing all video augmentation drops Video-MME to 45.7 and HRVideoBench to 44.5, versus 48.2 and 51.0 for full VISTA-Mantis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune on VISTA-400K | VideoLLaVA · Video-MME | average accuracy: 39.9 -> 43.7 | Helped |
| Finetune on VISTA-400K | Mantis-Idefics2 · Video-MME | average accuracy: 45.4 -> 48.2 | Helped |
| Finetune on VISTA-400K | LongVA · Video-MME | average accuracy: 52.4 -> 55.5 | Helped |
| Finetune on VISTA-400K | VideoLLaVA · HRVideoBench | average accuracy: 32.5 -> 47.5 | Helped |
| Finetune on VISTA-400K | Mantis-Idefics2 · HRVideoBench | average accuracy: 48.5 -> 51.0 | Helped |
| Finetune on VISTA-400K | LongVA · HRVideoBench | average accuracy: 48.0 -> 50.0 | Helped |
| Add Long Video Captioning subset (ablation) | VISTA-Mantis · Video-MME | average accuracy: 47.9 -> 48.2 | Helped |
| Add Event Relationship QA subset (ablation) | VISTA-Mantis · Video-MME | average accuracy: 47.7 -> 48.2 | Helped |
| Add Temporal NIAH subset (ablation) | VISTA-Mantis · Video-MME | average accuracy: 47.5 -> 48.2 | Helped |
| Add Two Needle NIAH subset (ablation) | VISTA-Mantis · Video-MME | average accuracy: 48.1 -> 48.2 | Helped |
| Add Spatial NIAH subset (ablation) | VISTA-Mantis · HRVideoBench | average accuracy: 47.5 -> 51.0 | Helped |
| Add Spatiotemporal NIAH subset (ablation) | VISTA-Mantis · HRVideoBench | average accuracy: 50.0 -> 51.0 | Helped |
Surprising findings
- Simply adding more input frames at test time does not reliably improve high-resolution video understanding, as Mantis-Idefics2 beats LongVA on HRVideoBench despite weaker long-video performance.
- Long-video augmentations also improve high-resolution video understanding and vice versa, suggesting transfer between the two capabilities.
Limitations
- Relies on Gemini-1.5-Pro for QA generation, so QA quality depends on a proprietary model.
- HRVideoBench has only 200 multiple-choice questions.
- Most augmented videos are under 3 minutes, and short-video gains are mixed (e.g., Mantis slightly drops on NExT-QA).
Experimental setup
Datasets: Panda-70M MiraData FineVideo InternVid OpenVid-1M VideoChat2-IT
Benchmarks: Video-MME MLVU LVBench LongVideoBench MVBench NExT-QA HRVideoBench MSVD-QA MSRVTT-QA TGIF-QA ActivityNet-QA
Metrics: accuracy score (1-5)
Baselines: VideoLLaVA Mantis-Idefics2 LongVA GPT-4V GPT-4o Gemini-1.5-Pro Video-XL LongVILA LongLLaVA
Builds on: CutMix MixUp VideoMix LLaVA-Video-178K
Closest papers by meaning
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.90)
- VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (CVPR 2025 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences (NeurIPS 2026 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- Apollo: An Exploration of Video Understanding in Large Multimodal Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.