Research map / Vision-language and multimodal / Multimodal LLMs

Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Multi-Crit, a benchmark of 425 multimodal prompts with multi-criterion human preference annotations to test whether LMM judges follow pluralistic criteria and resolve criterion-level conflicts, with three new metrics (PAcc, TOS, CMR).

Problem

LMM-as-a-judge benchmarks assign a single overall preference per response pair, so they cannot test whether multimodal judges follow diverse, fine-grained evaluation criteria or recognize criterion-level trade-offs and conflicts.

Approach

Annotate each response pair with separate human preferences per criterion (five open-ended and five reasoning criteria), keep pairs with intra-sample criterion conflicts, and score judges with pluralistic accuracy, trade-off sensitivity, and conflict matching rate.

Key results

• o4-mini achieves the highest open-ended pluralistic accuracy at 32.78% and reasoning pluralistic accuracy of 53.17%, with best overall reasoning average of 80.85%.
• Proprietary models' conflict matching rate drops about 9.4 points (43.1 to 33.7) in open-ended and 18.1 points (65.8 to 47.7) in reasoning when moving to open-source models.
• Test-time majority voting raises o4-mini's pluralistic accuracy from 32.78% to 37.12% (open-ended) and 53.17% to 57.94% (reasoning).

Surprising findings

• Fine-tuned critic models (LLaVA-Critic, R1-Reward, UnifiedReward) do not consistently outperform their base models on pluralistic criterion-level judgment, improving mainly on visual grounding.
• RL fine-tuning with holistic accuracy rewards on reasoning reduces trade-off sensitivity and conflict matching in reasoning judgment.

Limitations

• Small benchmark with 425 prompts and 1,425 criterion-level judgments, limiting statistical power.
• Judge prompt evaluates one criterion at a time, so it may not reflect holistic multi-criteria judgment settings.
• Candidate responses come from a fixed set of 11 LMMs and filtering uses GPT-4o, Gemini, and Claude judges, which may bias difficulty selection.

Experimental setup

Datasets: ImageInWords DOCCI WildVision-Bench WildVision-Battle MathVerse MM-K12 EMMA-mini VisualPuzzles

Benchmarks: Multi-Crit

Metrics: PAcc TOS CMR criterion-level accuracy Cohen's kappa

Baselines: o4-mini Claude-3.7-Sonnet GPT-5 GPT-4o Gemini-2.5-Pro Qwen2.5-VL-72B-Instruct InternVL3.5-38B LLaVA-Critic R1-Reward UnifiedReward

Builds on: MLLM-as-a-Judge VL-RewardBench Multimodal-RewardBench GRPO

Closest papers by meaning

• Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators (NeurIPS 2025 · similarity 0.91)
• LLaVA-Critic: Learning to Evaluate Multimodal Models (CVPR 2025 · similarity 0.89)
• MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.89)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.89)
• PhyCritic: Multimodal Critic Models for Physical AI (CVPR 2026 · similarity 0.88)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.88)