Research map / Vision-language and multimodal / Multimodal LLMs

How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks multimodal foundation models (GPT-4o, o4-mini, Gemini, Claude 3.5 Sonnet, Qwen2-VL, Llama 3.2) on standard vision tasks via API-compatible prompt chaining, with specialist and chained control baselines.

Problem

Vision-only understanding of multimodal foundation models is unclear because they output text, not dense outputs, and many are API-only; existing benchmarks are VQA-style and not comparable to vision specialists.

Approach

Decompose standard vision tasks (classification, detection, segmentation, grouping, depth, surface normals) into text-solvable classification sub-tasks via superpixels, recursive zooming, and pairwise ranking, so any image-in text-out MFM can be benchmarked against specialists with the same metrics and calibrated control baselines.

Key results

• GPT-4o achieves 77.20% ImageNet top-1 and 31.87 AP on COCO detection, the best non-reasoning MFM, while Model Soups ViT-G reaches 90.94% and Co-DETR 80.23 AP.
• On semantic segmentation GPT-4o reaches 44.89 mIoU versus 65.52 for OneFormer; on grouping GPT-4o reaches 59.06 mIoU versus 80.87 for SAM 2.
• On surface normals, o4-mini gets the best MFM rho_z (0.46) and rho_x (0.22), while most MFMs show weak or negative correlations; reasoning models o1, o3 and o4-mini show gains on geometric tasks.

Surprising findings

• Llama 3.2 90B performs well on segmentation even without image crops, surpassing other MFMs in that ablation.
• Models show a 'ceiling bias' in blind-guess depth, attributing greater depth to pixels higher in the image.

Limitations

• Prompt chaining requires many API calls per sample, making evaluation costly; o1 and o3 are evaluated only on subsets.
• The framework is a benchmark rather than an optimal method, so results depend on the chosen prompt chain granularity and may underestimate MFMs.
• Depth and normal predictions are relative and rely on superpixel assumptions and ground-truth scale alignment.

Experimental setup

Datasets: ImageNet ImageNet-v2 ImageNet-R ImageNet-Sketch ImageNet-C ImageNet-3DCC COCO Hypersim

Benchmarks: ImageNet ImageNet-v2 ImageNet-R ImageNet-Sketch ImageNet-C ImageNet-3DCC COCO Hypersim

Metrics: top-1 accuracy AP mIoU pixel accuracy Spearman correlation AbsRel delta thresholds pairwise accuracy

Baselines: Model Soups ViT-G OpenCLIP H Co-DETR DETR 4M-21 OneFormer SAM SAM 2 Omnidata Lotus MoGe-2 DSINE

Builds on: Cambrian-1 SAM Omnidata DetToolChain

Closest papers by meaning

• 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.88)
• GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.88)
• MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.87)
• Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.87)
• ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.86)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.86)