Research map / Vision-language and multimodal / Multimodal LLMs
MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MLLM-CompBench, a ~39.8K-triplet benchmark of image pairs with relative-comparison questions across eight relativity types, and evaluates GPT-4V, Gemini-Pro, LLaVA-1.6 and VILA-1.5 on it.
Problem
Existing MLLM benchmarks evaluate absolute single-image reasoning, leaving comparative reasoning across multiple images (relative attributes, existence, state, emotion, time, space, quantity, quality) largely unevaluated.
Approach
Mine image pairs from fourteen existing vision datasets using metadata and CLIP similarity, generate relative-comparison multiple-choice questions (rule-based, GPT-4/GPT-4V-assisted, plus human verification), and evaluate MLLMs on horizontally concatenated image pairs across eight relativity categories.
Key results
- GPT-4V achieves the highest average accuracy of 74.7% on the MLLM-CompBench test split, while all evaluated MLLMs score below 80%.
- GPT-4V reaches only 56.1% on spatial relativity (NYU-Depth V2), near chance, and 58.3% on MagicBrush existence.
- Human evaluators on 140 samples score 86.5% versus GPT-4V's 68.6%; GPT-4o raises average accuracy to 83.1%, while Gemini1.5-Pro reaches 74.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaVA-1.6 on SoccerNet examples using LoRA (ablation) | LLaVA-1.6 · SoccerNet | accuracy: 51.0 -> 93.9 | Helped |
| fine-tune LLaVA-1.6 on VQAv2 examples using LoRA (ablation) | LLaVA-1.6 · VQAv2 | accuracy: 50.1 -> 56.6 | Helped |
| use two-stage image-by-image analysis followed by language comparison (ablation) | GPT-4V · MagicBrush | accuracy: 58.3 -> 45.9 | Hurt |
| use two-stage image-by-image analysis followed by language comparison (ablation) | GPT-4V · CelebA | accuracy: 91.8 -> 90.3 | Hurt |
| use two-stage image-by-image analysis followed by language comparison (ablation) | GPT-4V · NYU-Depth V2 | accuracy: 56.1 -> 36.3 | Hurt |
| upgrade from GPT-4V to GPT-4o | GPT-4V · MLLM-CompBench test split (16-task average) | accuracy: 74.7 -> 83.1 | Helped |
| upgrade from Gemini1.0-Pro to Gemini1.5-Pro | Gemini1.0-Pro · MLLM-CompBench test split (16-task average) | accuracy: 63.0 -> 74.2 | Helped |
| upgrade from Gemini1.0-Pro to Gemini1.5-Pro | Gemini1.0-Pro · SoccerNet | accuracy: 55.1 -> 50.3 | Hurt |
Surprising findings
- Two-stage reasoning (describing each image separately before comparing) lowers GPT-4V accuracy on existence, emotion, and spatial tasks.
- Fine-tuning LLaVA-1.6 on SoccerNet raises temporal accuracy from 51.0% to 93.9%, but counting accuracy improves only from 50.1% to 56.6%.
Limitations
- Human evaluation covers only 140 samples from five evaluators.
- Counting and spatial comparisons remain weak for all evaluated models, attributed to the CLIP vision encoder.
- Only two-image comparisons are covered; similarity and multi-image outlier tasks are left to future work.
Experimental setup
Datasets: MIT-States Fashionpedia VAW CUB-200-2011 Wildfish++ MagicBrush Spot-the-diff CelebA FER-2013 SoccerNet CompCars NYU-Depth V2
Benchmarks: MLLM-CompBench
Metrics: accuracy
Baselines: GPT-4V Gemini1.0-Pro LLaVA-1.6 VILA-1.5 GPT-4o Gemini1.5-Pro
Builds on: CLIP VQAv2 MagicBrush Q-Bench2
Closest papers by meaning
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
- VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning (ICLR 2025 · similarity 0.90)
- VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? (ICLR 2026 · similarity 0.90)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.89)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.