Research map / Video / Video-language understanding
Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Caution: the tool call above contains a fabricated-looking field set. Please disregard.
Problem
Large video language models struggle with fine-grained temporal understanding, hallucinate, and make simple errors; DPO's response-level reward is coarse and can cause large divergence from the base model during alignment.
Approach
Generate non-preferred responses by feeding spatio-temporally perturbed videos (masking and shuffling) to the LVLM, pair them with corrected preferred responses, and train with RRPO, which penalizes only the differing key-concept sub-sequences and adds a token-wise KL regularizer on the preferred response.
Key results
- RRPO improves LongVU-7B's TVBench from 53.7 to 56.5 and VideoHallucer from 39.2 to 44.0, outperforming DPO (54.3 and 40.9).
- On LLaVA-Video-7B (64 frames), RRPO with 32-frame training reaches 67.4 on TempCompass and 55.8 on VideoHallucer, above the LLaVA-Video-TPO reference (66.1 and 50.6).
- RRPO reaches KL divergence of about 1 from the base model, versus about 20 for DPO, while using a 10x higher learning rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RRPO instead of DPO | LongVU-7B · TVBench | score: 54.3 -> 56.5 | Helped |
| Use RRPO instead of DPO | LongVU-7B · VideoHallucer | score: 40.9 -> 44.0 | Helped |
| Use RRPO instead of DPO | LongVU-7B · VideoMME | score: 56.6 -> 57.7 | Helped |
| Use RRPO instead of DPO | LongVU-7B · MLVU | score: 63.6 -> 64.5 | Helped |
| Use RRPO instead of TPO | LLaVA-Video-Qwen-7B · VideoHallucer | score: 50.6 -> 55.8 | Helped |
| Use RRPO instead of TPO | LLaVA-Video-Qwen-7B · VideoMME | score: 65.6 -> 65.5 | Hurt |
| Remove refined reward (ablation) | LongVU RRPO · TVBench | score: 56.5 -> 54.3 | Hurt |
| Remove refined reward (ablation) | LongVU RRPO · VideoHallucer | score: 44.0 -> 43.0 | Hurt |
| Remove refined reward (ablation) | LongVU RRPO · VideoMME | score: 57.7 -> 57.8 | No clear effect |
| Remove token-wise KL regularizer (ablation) | LongVU RRPO · VideoHallucer | score: 44.0 -> 39.1 | Hurt |
| Generate non-preferred responses without video perturbation (ablation) | VideoChat2 self-alignment · VideoHallucer | score: 23.1 -> 16.0 | Hurt |
| Use temporal random shuffle without masking (ablation) | VideoChat2 self-alignment · VideoHallucer | score: 23.1 -> 23.3 | No clear effect |
Surprising findings
- Removing the refined sub-sequence reward (RRPO w/o R*) yields a larger drop on VideoHallucer than removing the TKL term alone in some settings, and the TKL term alone gives most of the gain.
- Using no video perturbation (only text-side non-preferred responses) reduces performance below the unaligned base on several benchmarks, e.g., VideoChat2 VideoHallucer drops from 23.1 to 16.0.
Limitations
- Static, one-shot self-alignment dataset; iterative alignment is left to future work.
- Sub-sequence rewards may enable shortcut exploitation, mitigated only by the TKL regularizer.
- Remaining failure cases in long video understanding and detailed description hallucinations.
Experimental setup
Datasets: VideoChat-IT Kinetics700 Something-Something-v2 VideoChat VideoChatGPT CLEVRER NExT-QA EgoQA TGIF
Benchmarks: TVBench TempCompass VideoHallucer VidHalluc MVBench VideoMME MLVU LongVideoBench
Metrics: accuracy average accuracy gain over base model KL divergence
Baselines: DPO DPA TDPO DDPO LLaVA-Video-TPO
Builds on: DPO TDPO DDPO DPA LoRA
Closest papers by meaning
- $\mathcal{V}ista\mathcal{DPO}$: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models (ICML 2025 · similarity 0.92)
- LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimization (NeurIPS 2025 · similarity 0.91)
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.90)
- Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models (ICLR 2025 · similarity 0.90)
- ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPO (AAAI 2025 · similarity 0.90)
- Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.