Research map / Vision-language and multimodal / Multimodal LLMs

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PerceptionRubrics, a rubric-based benchmark of 1,038 information-dense images with about 10.7k atomic rubrics derived from consensus golden captions, scored with a gated (Must-Right) metric that calibrates MLLM evaluation to human preference.

Problem

Existing perception and captioning benchmarks are saturated, use sparse or biased references, and rely on linear averaging that hides fatal localized hallucinations, so they poorly reflect real-world perceptual reliability and human judgment.

Approach

Derive golden captions via a circular peer-review consensus among several MLLMs (with discard-on-divergence and human verification), distill them into Must-Right and Easy-Wrong atomic rubrics, and score with a multiplicative gate so failing any mandatory fact zeroes the score.

Key results

• Seed-2.0-Lite leads with 70.07% overall score; best open-source Qwen3.5-397B-A17B reaches 61.61%, over 8 points behind proprietary leaders.
• PerceptionRubrics correlates with Vision Arena at Pearson 0.916 and Spearman 1.000 on five overlapping models, outperforming DOCCI and DetailCaps.
• Models achieve high atomic accuracy but much lower Must-Right all-pass rate, a 'Reliability Gap' that narrows with model capability; Must-Right pass rate and Easy-Wrong accuracy show R^2 about 0.98.

Surprising findings

• GPT-4o-2024-05-13 scores only 12.59% overall, the weakest among evaluated models.
• Open-source models trail proprietary frontiers by over 8% on perception, unlike reasoning benchmarks where they rival flagships.

Limitations

• Judge is an LLM (GPT-OSS-120B) and rubrics are generated by MLLMs, so evaluation may inherit model biases despite human verification.
• Golden captions are discarded when models disagree, so the benchmark is limited to high-consensus samples.
• Human alignment validated on only five models, limiting the strength of the correlation claim.

Experimental setup

Datasets: open-source visual benchmarks (image sources) DOCCI DetailCaps-4870

Benchmarks: PerceptionRubrics Vision Arena (human preference alignment)

Metrics: gated score Must-Right pass rate Atomic accuracy Pearson correlation Spearman correlation standard deviation across runs

Baselines: DOCCI DetailCaps CLIPScore Vision Arena

Builds on: CLIPScore DOCCI DetailCaps RaR RM-R1

Closest papers by meaning

• The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.89)
• DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.89)
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.88)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception (NeurIPS 2025 · similarity 0.87)
• SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions (NeurIPS 2026 · similarity 0.87)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.87)