Research map / Vision-language and multimodal / Multimodal LLMs

Unveiling the Visual Counting Bottleneck in Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Diagnoses why VLMs fail to count beyond training distributions using a synthetic Go-board toy VLM and linear hidden-number probes, then validates on Qwen3-VL-32B, concluding the failure lies in symbolic mapping (a "fractured magnitude" between visual and textual number spaces).

Problem

VLMs interpolate well but fail to generalize visual counting to quantities beyond the training distribution; it is unclear whether the failure stems from perception, magnitude representation, or mapping to symbolic number tokens.

Approach

Decomposes visual counting into individuation, magnitude awareness, and symbolic mapping, and tests each with linear hidden-number probes, comparative counting, layer-wise probing, and attention-head circuit analysis under a decoupled vision/text curriculum, arguing the bottleneck is a fractured, modality-specific number space.

Key results

• Toy VLM visual counting collapses to 0% in the visual extrapolation regime (N=50-99) while textual counting remains at 100%.
• Linear probes trained on ID data recover hidden counts with near-zero vision gap across 0-120, while the language gap explodes at N=50.
• On Qwen3-VL-32B, visual counting collapses to near random for N>8 while textual counting stays robust; the model keeps >70% accuracy on comparative visual-vs-text matching in the VE regime.

Surprising findings

• The toy VLM retains >70% accuracy on comparing visual and text cardinalities in the VE regime despite 0% explicit enumeration accuracy.
• Visual and textual counting use almost entirely disjoint attention-head sets (73.58% non-overlap).

Limitations

• Real-world validation is limited to a single model (Qwen3-VL-32B) and a simplified Go-board task with identical objects.
• Perceptual failures from occlusion and semantic filtering in natural images are not studied.
• Causal and probing evidence is correlational in parts; the 'fractured' interpretation rests on attention-head overlap and probes.

Experimental setup

Datasets: synthetic Go board images (19x19 for toy VLM, 6x6 for Qwen3-VL evaluation) synthetic text character-counting sequences

Benchmarks: synthetic Go board visual counting (ID 0-49, VE 50-99, FE 100-120) synthetic text counting Qwen3-VL 6x6 Go board counting

Metrics: exact match accuracy vision gap |N_H - N_G| language gap |N_H - N_P| steering accuracy probe accuracy

Baselines: textual counting (same toy VLM)

Builds on: M-RoPE

Closest papers by meaning

• Understanding Counting Mechanisms in Large Language and Vision-Language Models (CVPR 2026 · similarity 0.92)
• Vision Language Models are Biased (ICLR 2026 · similarity 0.90)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
• Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.88)
• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.88)
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.88)