Research map / Vision-language and multimodal / Multimodal LLMs
ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ISR-DPO, an iterative self-rewarding DPO for video LMMs where the VLMM judges its own response pairs conditioned on a self-retrospectively refined video description, reducing length bias and verbosity hallucination.
Problem
Iterative self-rewarding preference optimization for video large multimodal models suffers from modality misalignment in the self-judge (favoring linguistic priors) and length bias that induces visually ungrounded verbose responses.
Approach
The self-judge is conditioned on a visual context (detailed video description) that is iteratively refined from the previous iteration's context (self-retrospection), so preference selection is grounded in video content rather than length or language priors.
Key results
- On in-domain benchmarks, ISR-DPO (iteration 9) reports ActivityNet-QA 82.99% accuracy vs 76.62% for LLaVA-Hound-DPO.
- On out-domain MSVD-QA, MSRVTT-QA, TGIF-QA and SSV2-QA, ISR-DPO iteration 9 reaches 80.36, 75.42, 78.58 and 54.66 accuracy, above LLaVA-Hound-DPO.
- Human alignment accuracy of preference selection rises from 59.0% (self-rewarding) to 75.0% (ISR-DPO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-retrospective visual context to VLMM-as-a-judge preference selection (vs self-rewarding baseline) | iterative DPO for VLMM · human preference alignment | human alignment accuracy (%): 59.0 -> 75.0 | Helped |
| use self-retrospective context instead of no context (ablation) | preference selection in iterative DPO · MSVD-QA out-domain | Acc.: 78.73 -> 80.36 | Helped |
| use self-retrospective context instead of no context (ablation) | preference selection in iterative DPO · SSV2-QA out-domain | Acc.: 54.34 -> 54.66 | Helped |
| reuse context from first iteration (Fixed) instead of no context (ablation) | preference selection in iterative DPO · MSVD-QA out-domain | Acc.: 78.73 -> 79.17 | Helped |
| renew context each iteration instead of no context (ablation) | preference selection in iterative DPO · SSV2-QA out-domain | Acc.: 54.34 -> 53.03 | Hurt |
| renew context each iteration instead of no context (ablation) | preference selection in iterative DPO · MSRVTT-QA out-domain | Acc.: 73.42 -> 74.04 | Helped |
| iterate self-retrospective DPO for 9 rounds instead of first iteration | ISR-DPO · ActivityNet-QA in-domain | Acc.: 75.58 -> 82.99 | Helped |
| iterate self-retrospective DPO for 9 rounds instead of first iteration | ISR-DPO · MSRVTT-QA out-domain | Acc.: 68.18 -> 75.42 | Helped |
| ISR-DPO final model vs LLaVA-Hound-DPO on in-domain benchmarks | ISR-DPO · ActivityNet-QA / VIDAL-QA / WebVid-QA in-domain | Acc.: +6.37 / +8.94 / +8.29 Acc. over LLaVA-Hound-DPO | Helped |
| ISR-DPO final model vs prior models on out-domain concise-answer benchmarks | ISR-DPO · MSVD / MSRVTT / TGIF / SSv2 out-domain (Table 2) | Acc.: MSVD 73.64->80.36; MSRVTT 68.29->75.42; TGIF 74.00->78.58; SSv2 48.89->54.66 | Helped |
| ISR-DPO final model vs baselines on GPT-3.5 evaluated out-domain QA | ISR-DPO · MSVD / MSRVTT / TGIF (Table 3) | Acc.: MSVD 81.0->85.8; MSRVTT 70.2->78.7; TGIF 62.3->67.8 | Helped |
Surprising findings
- Renewing visual context each iteration performs worse than using a fixed first-iteration context on most benchmarks, and worse than no context on SSv2 and WebVid.
Limitations
- Relies on self-generated context and judgments; errors can propagate across iterations.
- Human alignment evaluation uses only 100 questions and 15 annotators per question.
Experimental setup
Datasets: LLaVA-Hound video-instruction data (17k pairs) ActivityNet-QA VIDAL-QA WebVid-QA MSVD-QA MSRVTT-QA TGIF-QA SSV2-QA
Benchmarks: ActivityNet-QA VIDAL-QA WebVid-QA MSVD-QA MSRVTT-QA TGIF-QA SSV2-QA
Metrics: accuracy score win-rate human alignment accuracy average response word length
Baselines: Video-ChatGPT LLaMA-VID Chat-UniVi Video-LLaVA VLM-RLAIF PLLaVA LLaVA-NeXT-DPO LLaVA-Hound-DPO self-rewarding
Builds on: DPO self-rewarding language models LLaVA-Hound-DPO
Closest papers by meaning
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.90)
- Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization (ICLR 2026 · similarity 0.88)
- Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.88)
- MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (ICLR 2025 · similarity 0.87)
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.