Research map / Vision-language and multimodal / Multimodal LLMs

Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MeasureBench, a 2,442-item benchmark of real and procedurally synthesized measuring-instrument images, shows frontier VLMs struggle at value reading, and finds GRPO on synthetic gauge data improves both synthetic and real-world accuracy.

Problem

Vision-language models struggle to read measurement instruments (dials, digital, linear, composite gauges), which requires fine-grained pointer/scale localization and mapping to numeric values; no broad benchmark existed for this.

Approach

A benchmark spanning 26 real instrument types plus a hybrid 2D-programmatic and 3D-Blender synthesis pipeline that generates instrument images with precisely known readings, which are then used for rule-based GRPO reinforcement finetuning.

Key results

• Gemini-2.5-Pro is best with 30.2% overall accuracy on real-world images and 26.3% on synthetic images, while unit accuracy exceeds 90% for most models.
• GRPO on 3,900 synthetic images raises Qwen2.5-VL-7B from 14.6% to 19.7% on real-world images and from 10.9% to 35.2% on the synthetic subset.
• SFT on the same synthetic data improves synthetic accuracy (29.7%) but degrades real-world accuracy (12.5% answer SFT, 8.8% rationale SFT), while GRPO improves both.

Surprising findings

• Enabling thinking yields little improvement and sometimes degrades accuracy on instrument reading.
• Qwen2.5-VL-72B answers '10:10' on 72.88% of real-world clock images, suggesting a strong language prior from training data.

Limitations

• Real-world gains from RFT are modest and the synthetic data does not transfer strongly to out-of-distribution instruments.
• Composite instruments remain near 0% accuracy for most models, and reasoning gains from thinking are limited.
• Accuracy can be inflated by guessing and error cancellation, which final-answer scoring does not detect.

Experimental setup

Datasets: MeasureBench real-world subset MeasureBench synthetic subset

Benchmarks: MeasureBench real-world MeasureBench synthetic MMMU MMMU-Pro MathVista TextVQA

Metrics: overall accuracy value accuracy unit accuracy

Baselines: Gemini-2.5-Pro GPT-5 GPT-5-Mini Qwen3-VL-235B Qwen2.5-VL-72B Claude-Opus-4.1 Reitsma et al. Shu et al.

Builds on: GRPO verl FlagEvalMM DeepSeek-R1

Closest papers by meaning

• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
• SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.88)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.88)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)
• Expand VSR Benchmark for VLLM to Expertize in Spatial Rules (AAAI 2025 · similarity 0.88)