Research map / Video / Video-language understanding
Vript: A Video Is Worth Thousands of Words
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Vript, a densely annotated video-text dataset of ~420K script-style clips with camera-language captions, trains the Vriptor captioner on ST-LLM, and introduces the Vript-Hard benchmark (HAL, RR, ERO).
Problem
Existing video-text datasets have short, coarse captions that document only static content and ignore camera operations, and video understanding benchmarks are too short and superficial to test hallucination and long-video reasoning.
Approach
Annotate untrimmed videos scene-by-scene in a video-script format that records both content and camera language (shot type, camera movement), then align more text with video via script concatenation, voice-over input, and timestamps; pair this with a hard benchmark of hallucination, retrieval-then-reasoning, and event re-ordering tasks.
Key results
- Vriptor-S/voice reaches F1 58.3 on Vript-HAL versus 49.3 for GPT-4V and 49.6 for GPT-4O (1 fps/100 fr).
- On Vript-RR, GPT-4O (1 fps/100 fr) scores 91.4 / 78.2 (scene multiple-choice / open-ended) and GPT-4V scores 80.9 / 75.0, while the best open-source model VILA-1.5 8B scores 75.0 / 48.7.
- On Vript-ERO, the best @3 score is 38.6 for GPT-4O (1 fps/100 fr), and most models give correct order for all three scenes in only about 20% of questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use scene-by-scene captions instead of whole-video descriptions (ablation) | Vriptor · Vript-HAL | recall | Helped |
| Use scene-by-scene captions instead of whole-video descriptions (ablation) | Vriptor · Vript-HAL | precision | Hurt |
| Add voice-over transcription to whole-video captioning (ablation) | Vriptor-W · Vript-HAL | precision: 79.1 -> 80.3 | Helped |
| Add voice-over transcription to whole-video captioning (ablation) | Vriptor-W · Vript-HAL | recall: 26.8 -> 27.7 | Helped |
| Add voice-over transcription to whole-video captioning (ablation) | Vriptor-W · Vript-HAL | F1: 40.0 -> 41.1 | Helped |
| Add voice-over transcription to scene-by-scene captioning (ablation) | Vriptor-S · Vript-HAL | recall: 46.1 -> 47.2 | Helped |
| Add voice-over transcription to scene-by-scene captioning (ablation) | Vriptor-S · Vript-HAL | F1: 56.6 -> 58.3 | Helped |
| Increase the scene count from 2 to 5 (ablation) | Vriptor-S · Vript-HAL | recall: 40.9 -> 57.5 | Helped |
| Increase the scene count from 2 to 5 (ablation) | Vriptor-S · Vript-HAL | precision: 75.8 -> 71.4 | Hurt |
| Add timestamps to video-script alignment (ablation) | Vriptor · Vript-HAL | recall: +12% | Helped |
| Use GPT-4O at 1 fps or up to 100 frames rather than its standard setting (ablation) | GPT-4O · Vript-ERO | @3 accuracy: 32.6 -> 38.6 | Helped |
Surprising findings
- Models such as BLIP2 and VideoChat2 score low on hallucination only because they produce shorter captions with fewer details, so precision alone is misleading.
- Performance on Vript-RR drops for scenes in the middle of the video (40% and 60%) even though the visual token count is well below the length where text needle-in-a-haystack drops are usually seen.
Limitations
- Vript annotations were generated by GPT-4V and were not manually checked, so hallucinations may remain.
- Vript-Hard ground truth was manually revised, but the released dataset may still contain errors and the scale of the benchmark is limited.
- Vriptor's training data and base model licenses restrict use to academic research only.
Experimental setup
Datasets: Vript HD-VILA-100M Vript-Extend MSR-VTT
Benchmarks: Vript-HAL Vript-RR Vript-ERO MSR-VTT
Metrics: precision recall F1 accuracy BLEU ROUGE CIDEr
Baselines: VideoChatGPT Video-LLaMA VideoChat VideoChat2 ST-LLM PLLaVA 7B VILA-1.5 8B GPT-4V GPT-4O Claude 3-Opus Claude 3.5-Sonnet Gemini-1.5-Pro
Builds on: ST-LLM GPT-4V annotation Whisper transcription PySceneDetect
Closest papers by meaning
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.89)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.89)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.89)
- Video ReCap: Recursive Captioning of Hour-Long Videos (CVPR 2024 · similarity 0.88)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.