Research map / Video / Video-language understanding
Is Your Video Language Model a Reliable Judge?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether aggregating judgments from multiple video language models (VLMs), including unreliable ones, improves VLM-as-judge reliability on CVRR-ES, using LLM agent-debate with reference answers as the reference standard.
Problem
Reliability of video language models as automatic judges of other VLMs' responses is underexplored; single VLM judges may be biased or hallucinate, and it is unclear whether ensembling/collective judgments from mixed-reliability pools helps.
Approach
Measures each VLM judge's agreement (weighted Cohen's kappa) with a reference-guided multi-LLM agent-debate evaluator, then tests collective-thought aggregation (GPT-4o final judge over all VLM reviews) and a reliability-based mixture of judges; finds mixing unreliable judges adds noise and fine-tuning Video-LLaVA does not fix it.
Key results
- GPT-4o is the only judge with substantial agreement with Agent-Debate (e.g., 60.38 kappa on Social Context, 33.11 average across dimensions), while Video-LLaVA and LLaMA-VID show low or negative agreement.
- Collective thought with all judges yields average agreement of about 2.72 to 42.53 across dimensions, which does not beat GPT-4o alone.
- Mixture of judges selected by per-dimension kappa does not substantially improve agreement (averages up to ~56 on Social Context, comparable to collective thought); fine-tuning Video-LLaVA only modestly improves its kappa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate reviews from all initial judges with GPT-4o as final judge | GPT-4o as a sole judge · CVRR-ES, Social Context | weighted Cohen's kappa (%): 60.38 -> 42.53 | Hurt |
| Select initial judges by per-dimension reliability, with GPT-4o as final judge | GPT-4o as a sole judge · CVRR-ES, Social Context | weighted Cohen's kappa (%): 60.38 -> 56.23 | No clear effect |
| Aggregate reviews from all initial judges with GPT-4o as final judge | GPT-4o as a sole judge · CVRR-ES, Non-exist (NE) | weighted Cohen's kappa (%): 23.09 -> 2.72 | Hurt |
| Select initial judges by per-dimension reliability, with GPT-4o as final judge | GPT-4o as a sole judge · CVRR-ES, Non-exist (NE) | weighted Cohen's kappa (%): 23.09 -> 2.72 | Hurt |
| Use GPT-4o rather than Video-LLaVA as judge | VLM judging on the VideoChatGPT dataset · VideoChatGPT dataset, average across candidates | weighted Cohen's kappa (%): 1.72 -> 44.42 | Helped |
| Fine-tune Video-LLaVA (ablation) | Video-LLaVA as a judge · CVRR-ES | weighted Cohen's kappa | No clear effect |
| Fine-tune Video-LLaVA (ablation) | Video-LLaVA as a judge · CVRR-ES | rating distribution | No clear effect |
Surprising findings
- Adding less reliable VLM judges to a collective pool makes the GPT-4o final judge perform worse than GPT-4o alone.
- Video-LLaVA rates nearly all candidates close to 4 and yields negative or near-zero agreement with the reference-guided debate.
Limitations
- Experiments limited to two datasets (CVRR-ES and a 1,000-sample VideoChatGPT subset) and a small set of VLMs and LLMs.
- Agent-Debate with reference answers is assumed to be the most reliable reference without human validation; reliance on weighted kappa may miss qualitative nuances.
- Relies on costly advanced models such as GPT-4o, limiting scalability.
Experimental setup
Datasets: CVRR-ES VideoChatGPT generic benchmark (1,000 samples used)
Benchmarks: CVRR-ES VideoChatGPT dataset
Metrics: weighted Cohen's kappa average rating score rating distribution
Baselines: GPT-4o Agent-Debate GPT-3.5 InternVL2 Video-LLaVA LLaMA-VID
Builds on: CVRR-ES wisdom of crowds weak-to-strong generalization
Closest papers by meaning
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.89)
- UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (NeurIPS 2025 · similarity 0.88)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.88)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.87)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.86)
- Beyond Accuracy: On the Effects of Fine-Tuning Towards Vision-Language Model’s Prediction Rationality (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.