Research map / Vision-language and multimodal / Multimodal LLMs
UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UniBench, a unified codebase implementing 53 VLM benchmarks and evaluating ~59 open VLMs zero-shot, finding scale helps little for relations and reasoning and that VLMs struggle on digit recognition and counting.
Problem
Vision-language model evaluation is fragmented across many benchmarks that are costly to implement and run, leaving blind spots and making cross-model comparison hard.
Approach
A unified, extensible evaluation framework covering 53 benchmarks grouped into seven types and seventeen capabilities, plus a distilled subset of representative benchmarks that runs in about 5 minutes on one A100, used to systematically analyze how scale, data quality and learning objectives affect VLM capabilities.
Key results
- Across 59 open VLMs, scaling training data by about 1000x and model size by about 11x yields only a 3.41% gain on relation and reasoning benchmarks versus 17.4% on other benchmark types.
- NegCLIP ViT B 32 (86M parameters) reaches 70.4% overall versus 50.5% for the 4.3B-parameter EVA ViT-E/14.
- VLMs barely reach 90% top-5 accuracy on MNIST, while a simple 2-layer MLP achieves 99% top-1 accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale model size from 86 million to 1 billion parameters | VLMs · Object recognition and other benchmark types | performance: +17.4% | Helped |
| scale model size from 86 million to 1 billion parameters | VLMs · Relations and reasoning | performance: +3.41% | Helped |
| increase training dataset size by 1000× | VLMs · Relations and reasoning | performance | No clear effect |
| use stricter CLIP-score filtering in training data | VLMs · Overall benchmark performance | performance | Helped |
| use a hard-negative learning objective | NegCLIP · Relational understanding | performance: 70.4% vs 50.5% | Helped |
| use specialized word-digit prompts instead of less specialized prompts (ablation) | VLMs · MNIST | accuracy | Helped |
Surprising findings
- Many of the best VLMs score near or below chance on simple tasks such as MNIST, SVHN, and counting despite training on billions of samples.
- ImageNet accuracy is poorly or negatively correlated with 18 of the 53 benchmarks.
Limitations
- Only covers standard zero-shot evaluation protocol.
- Cannot cover all existing or future VLMs and benchmarks.
- Relation benchmarks and some capabilities have few classes or weak correlations, e.g., DSPR position correlation of 0.29.
Experimental setup
Datasets: LAION-400M DataComp LAION-2B MetaCLIP Flickr30k COCO PMD
Benchmarks: ImageNet ImageNet-A ImageNet-O ImageNet-R ImageNet-Sketch ImageNet-V2 ObjectNet CIFAR-10 CIFAR-100 MNIST SVHN CLEVR
Metrics: zero-shot accuracy top-k accuracy median performance correlation
Baselines: NegCLIP EVA02 ViT E 14 DataComp ViT B 32 MNIST 2-layer MLP
Builds on: CLIP NegCLIP
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.92)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.91)
- VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.91)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.91)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.91)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.