Research map / Video / Video-language understanding
Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes alignment for answerability (SFT and DPO on a new UVQA dataset of unanswerable video questions built from MOMA-LRG and DiDeMo) so Video-LLMs refuse questions beyond the video's content, with new alignment metrics.
Problem
Video-LLMs are trained only on questions generated from video content, so they attempt to answer questions that the video cannot support and hallucinate responses instead of declining.
Approach
Equip Video-LLMs with answerability judgment by creating unanswerable questions via controlled alterations of scene descriptions (object, relation, attribute changes), training with SFT or DPO alongside answerable data, and measuring before/after behavior shifts with new refusal, permissiveness, and discretion scores.
Key results
- DPO-aligned VideoChat2 raises alignment score from 0.33 to 0.69 and accuracy from 0.22 to 0.54 on the balanced UVQA+ActivityNet-QA evaluation.
- DPO aligned VLM-RLAIF reaches alignment score 0.69 and accuracy 0.53 versus 0.25 unaligned; SFT gives 0.65 alignment.
- Prompting alone barely helps (VLM-RLAIF alignment 0.33 to 0.35), and POPE-style existence splitting is costlier and underperforms the trained model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align Video-LLaVA with answerability SFT | Video-LLaVA · UVQA + ActivityNet QA | S_align: 0.33 -> 0.66 | Helped |
| Align Video-LLaVA with answerability DPO | Video-LLaVA · UVQA + ActivityNet QA | S_align: 0.33 -> 0.68 | Helped |
| Align VideoChat2 with answerability SFT | VideoChat2 · UVQA + ActivityNet QA | S_align: 0.33 -> 0.62 | Helped |
| Align VideoChat2 with answerability DPO | VideoChat2 · UVQA + ActivityNet QA | S_align: 0.33 -> 0.69 | Helped |
| Align VLM-RLAIF with answerability SFT | VLM-RLAIF · UVQA + ActivityNet QA | S_align: 0.33 -> 0.65 | Helped |
| Align VLM-RLAIF with answerability DPO | VLM-RLAIF · UVQA + ActivityNet QA | S_align: 0.33 -> 0.69 | Helped |
| Align VLM-RLAIF with answerability DPO (ablation) | VLM-RLAIF (7B) · UVQA + ActivityNet QA | S_align: 0.33 -> 0.35 | No clear effect |
| Align Video-LLaVA with answerability DPO | Video-LLaVA · MSR-VTT out-of-distribution | S_align: 0.33 -> 0.58 | Helped |
| Align VLM-RLAIF with answerability DPO | VLM-RLAIF · MSR-VTT out-of-distribution | S_align: 0.33 -> 0.63 | Helped |
| Use DPO rather than SFT for answerability alignment | VLM-RLAIF · UVQA + ActivityNet QA | S_ex-ref: 0.10 -> 0.08 | Helped |
| Use DPO rather than SFT for answerability alignment | VLM-RLAIF · UVQA + ActivityNet QA | S_disc: 0.67 -> 0.64 | Mixed |
| Prompt the model to refuse unanswerable questions, without alignment training (ablation) | LLaVA-Video-Qwen2 (7B) · UVQA + ActivityNet QA | S_align: 0.33 -> 0.34 | No clear effect |
Surprising findings
- Scaling LLaVA-Video-Qwen2 from 7B to 72B does not meaningfully improve unanswerable-question performance.
- Explicit answerability prompting gives little to no benefit, even for the best open-source Video-LLMs.
Limitations
- Aligned models show increased excessive refusal on answerable questions, a trade-off between alignment and over-caution.
- Attribute-related unanswerable questions remain notably harder for all models.
- Evaluation relies on GPT-4o as judge for answer type labeling.
Experimental setup
Datasets: MOMA-LRG DiDeMo Video-ChatGPT ActivityNet-QA MSR-VTT
Benchmarks: UVQA evaluation set ActivityNet-QA subset MSR-VTT out-of-distribution set
Metrics: Excessive Refusal score Permissiveness score Discretion score alignment score answerability F1 accuracy LLM_score
Baselines: unaligned Video-LLaVA unaligned VideoChat2 unaligned VLM-RLAIF POPE-style existence splitting prompt-aligned VLM-RLAIF prompt-aligned LLaVA-Video-Qwen2
Builds on: DPO POPE Video-ChatGPT VLM-RLAIF
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.90)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.89)
- On the Consistency of Video Large Language Models in Temporal Comprehension (CVPR 2025 · similarity 0.89)
- Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.89)
- AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.