Research map / Vision-language and multimodal / Visual question answering

UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation

CVPR 2025 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UPME, an unsupervised peer-review framework where MLLMs generate questions from images, judge each other's answers with a vision-language score (correctness, visual reasoning, CLIP), and optimize model weights to estimate a leaderboard without human QA pairs.

Problem

Evaluating MLLMs requires costly human-designed VQA Q&A pairs, while MLLM-as-a-judge methods suffer from verbosity and self-preference biases and still rely on human-designed questions.

Approach

Models act as both question generators and peer judges on image-only data; a vision-language judgment score (response correctness, visual understanding and reasoning, and CLIP image-text correlation) is combined with confidence weights that are iteratively optimized with an MSE consistency loss to produce model scores.

Key results

• UPME reaches Pearson 0.944 and Spearman 0.972 with human-annotated MMStar scores, versus 0.878 Pearson for the best single judge GPT-4o.
• On ScienceQA, UPME reaches Pearson 0.814 and Spearman 0.886, versus 0.617 Pearson for GPT-4o and 0.463 for the original peer review.
• Human agreement on MMStar rises from 71.1% (peer review) to 95.9% (UPME); self-preference chi-square drops from 39.584 to 3.489.

Measured effects reported in the paper

Surprising findings

• The original peer review yields lower correlation with human scores than the single strong judge GPT-4o on MMStar, suggesting weaker models bias peer review.
• ScienceQA correlations are much lower than MMStar for all methods, which the authors attribute to limited visual dependency of ScienceQA.

Limitations

• Results on ScienceQA are considerably lower (Pearson 0.814), and the authors note the dataset may lack visual dependency.
• Relies on human-annotated benchmark scores (MMStar, ScienceQA) as the reference for correlation, so the alignment target is itself a benchmark.
• Review model may fail to answer the questions it generates or the original questions (disability of review model).

Experimental setup

Datasets: MMStar ScienceQA

Benchmarks: MMStar ScienceQA

Metrics: Pearson correlation Spearman correlation permutation entropy human agreement human consistency chi-square verbosity and self-preference bias tests

Baselines: Peer Review Majority Vote Rating Vote PRD individual MLLMs as judges

Builds on: LLM-as-a-Judge MT-Bench PRD peer rank and discussion

Closest papers by meaning

• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.90)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.90)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
• Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.90)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.89)