Research map / Video / Video-language understanding
$\mathcal{V}ista\mathcal{DPO}$: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VistaDPO, a hierarchical spatial-temporal direct preference optimization framework (instance, temporal, perceptive levels) plus a 7.2K-pair grounded video preference dataset, applied to Video-LLaVA and PLLaVA 7B.
Problem
Large video models suffer from misalignment with human intuition and video hallucination; existing video DPO (Hound-DPO) only aligns at the coarse instance/response level and ignores temporal and spatial visual preferences.
Approach
Extends DPO to video with three levels of preference optimization: response-level with relevant and irrelevant rejected responses plus video-level negatives, clip-level temporal negatives, and object-level spatial negatives with token-level sequential KL (TDPO-style) loss, trained on a newly built grounded dataset.
Key results
- On VideoHallucer overall, VistaDPO reaches 57.8 on PLLaVA (vs 38.1 base, 36.2 Hound-DPO) and 54.3 on Video-LLaVA (vs 17.8 base, 29.5 Hound-DPO).
- On MSVD/MSR-VTT QA with Video-LLaVA, VistaDPO scores 85.3/76.9 vs 71.8/59.0 for the base model.
- Ablation on VideoHallucer shows removing all levels except response-level drops overall accuracy from 54.3 to 39.8 (only L_DPO_r).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · VideoHallucer | Overall: 29.5 -> 54.3 | Helped |
| apply VistaDPO instead of Hound-DPO | PLLaVA · VideoHallucer | Overall: 36.2 -> 57.8 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · EventHallusion | Overall Binary: 33.3 -> 67.2 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · EventHallusion | Overall Descriptive: 9.5 -> 12.1 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · MSVD-QA | score: 80.7 -> 85.3 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · MSR-VTT-QA | score: 70.2 -> 76.9 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · TGIF-QA | score: 61.4 -> 74.1 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · ActivityNet-QA | score: 40.9 -> 55.0 | Helped |
| apply VistaDPO instead of Hound-DPO | Video-LLaVA · VideoChatGPT-Bench | Temporal: 2.0 -> 2.6 | Helped |
| add clip-level loss (ablation) | VistaDPO on Video-LLaVA · VideoHallucer | Overall: 53.0 -> 54.3 | Helped |
| add object-level loss (ablation) | VistaDPO on Video-LLaVA · VideoHallucer | Overall: 52.8 -> 54.3 | Helped |
| add object-level and token-level losses (ablation) | VistaDPO on Video-LLaVA · VideoHallucer | Overall: 49.4 -> 54.3 | Helped |
Surprising findings
- Hound-DPO reduces accuracy on the Basic category of VideoHallucer for PLLaVA (75.1 to 69.3) and on Desc. scores in EventHallusion.
- Reversing video frames as video-level negatives outperforms random or blackened negatives in the analysis.
Limitations
- Performance on long-duration videos with complex temporal dependencies is limited.
- Relies on manual annotation of keyframes and bounding boxes, which limits scale.
- Evaluation of several reference LVMs is not directly comparable due to different base models and data.
Experimental setup
Datasets: VistaDPO-7k MSR-VTT STAR VATEX ActivityNet-QA NExT-QA CLEVRER RoadTextVQA ViteVQA MSVD TempCompass Tomato
Benchmarks: VideoHallucer EventHallusion MSVD-QA MSR-VTT-QA TGIF-QA ActivityNet-QA VideoChatGPT-Bench MVBench
Metrics: accuracy binary accuracy description score open-ended QA accuracy captioning score (correctness, detail, context, temporal, co
Baselines: Hound-DPO Video-LLaVA PLLaVA VideoChatGPT VideoChat2 LLaMA-VID Video-LLaMA LLaMA-Adapter
Builds on: DPO Hound-DPO TDPO Video-LLaVA PLLaVA
Closest papers by meaning
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.92)
- VideoDPO: Omni-Preference Alignment for Video Diffusion Generation (CVPR 2025 · similarity 0.89)
- DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models (NeurIPS 2025 · similarity 0.89)
- LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimization (NeurIPS 2025 · similarity 0.88)
- VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens (CVPR 2024 · similarity 0.88)
- SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.