Research map / Vision-language and multimodal / Multimodal LLMs

GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GePBench, a synthetic multiple-choice benchmark (80K figures, 285K questions) for geometric perception in MLLMs, evaluates 32 models, and shows that GeP-data training improves LLaVA-1.5 and Qwen3-VL on downstream tasks.

Problem

MLLMs' ability to recognize geometric shapes and their spatial relationships (geometric perception) has not been systematically evaluated, as existing geometry datasets focus on mathematical reasoning.

Approach

A template-based data synthesis engine generates structured textual descriptions of geometric figures, renders them with noise, and builds multiple-choice questions across six perception aspects (existence, counting, location, size, reference, relationship), with the same data reused as training data (LLaVA-GeP) to improve downstream performance.

Key results

• GPT-5.2 scores only 29.0% on easy-split size estimation, and Qwen3-VL-235B-A22B reaches 54.9%, versus 99.3% human average accuracy.
• Qwen3.5-397B-A17B achieves the highest average (79.2%) among evaluated models, while most smaller open models fall below 70%.
• LLaVA-GeP-7B shows average gains of 2.1% on medical tasks and 2.2% on chart and document understanding over the LLaVA-1.5 baseline trained under the same settings.

Surprising findings

• Specialized geometry/math models (G-LLaVA, Math-PUMA, QVQ-preview) underperform their general base models on GePBench.
• GPT-4o overestimates shape size in 94% of size predictions, indicating systematic perceptual bias.

Limitations

• Benchmark consists of synthetic geometric figures with template-based questions, which may not reflect real-world image distributions.
• Training of GeP-enhanced models was done with LoRA and compared only on LLaVA-1.5-7B and Qwen3-VL-8B; the Qwen3-VL variant uses LLaVA-1.5 training data.
• Some baseline models were evaluated only on checkpoints from HuggingFace and the mixed encoder findings are attributed to speculation.

Experimental setup

Datasets: GePBench-train LLaVA pretraining data (LAION-CC-SBU 558K) LLaVA instruction tuning data

Benchmarks: GePBench VQAv2 GQA VizWiz TextVQA POPE MMBench SeedBench-Image MME MM-Vet MMMU DocVQA

Metrics: accuracy

Baselines: GPT-4o GPT-5.2 Gemini-2.5-pro Qwen3-VL Qwen3.5 InternVL3 G-LLaVA Math-LLaVA Math-PUMA QVQ-preview

Builds on: LLaVA LLaVA-1.5 Matplotlib

Closest papers by meaning

• GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning (CVPR 2026 · similarity 0.90)
• Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness (NeurIPS 2025 · similarity 0.88)
• GeoSense: Internalizing Geometric Necessity Perception for Multimodal Reasoning (ICML 2026 · similarity 0.88)
• GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation (ICLR 2026 · similarity 0.88)
• GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models (CVPR 2026 · similarity 0.88)