Research map / Vision-language and multimodal / Multimodal LLMs

Understanding Counting Mechanisms in Large Language and Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how LLMs and LVLMs represent and compute counts using controlled repeated-item tasks, activation patching, and a new causal probing tool, CountScope, revealing latent counters, separator shortcuts, and layerwise emergence.

Problem

Counting is a basic capability of LLMs and LVLMs, yet the internal mechanisms by which these models represent, update, and output numerical information during counting remain poorly understood; prior work mostly measures behavioral accuracy.

Approach

Introduces CountScope, a minimal activation-patching probe that transfers activations from source contexts into a fixed placeholder-plus-question target to decode latent count information, and uses it with a Causal Influence score to localize and characterize internal counters across text and vision inputs.

Key results

• Offline zero patching of context tokens drops ground-truth probability to 0.73 for LLM and 0.65 for LVLM, versus 0.03 and 0.05 for question tokens.
• Separator patching causes accuracy drops of 0.75 (monotypic) and 0.97 (polytypic), indicating models rely on separators as shortcuts.
• Counting accuracy for 10-20 objects rises from 10.0% baseline to 50.4% with spatial partitioning, versus 15.3% with CoT without partitions.

Surprising findings

• Separator tokens alone carry more predictive count information than element tokens in polytypic settings.
• Background visual patches often carry stronger count signals than foreground object patches.

Limitations

• Experiments focus on Qwen2.5 7B and Qwen2.5-VL 7B in main text; other models deferred to appendix.
• Evaluation relies on synthetic datasets with simple fruit lists and geometric shapes; natural data only minimally analyzed.
• No complete circuit discovery; CoT-based counting not fully understood.

Experimental setup

Datasets: synthetic textual fruit lists synthetic geometric shape images

Benchmarks: synthetic textual counting dataset synthetic visual counting dataset

Metrics: softmax probability of ground-truth count Causal Influence score accuracy cosine similarity PCA projection

Baselines: Logit Lens Tuned Lens chain-of-thought

Builds on: activation patching Patchscopes Logit Lens Tuned Lens causal mediation analysis

Closest papers by meaning

• Unveiling the Visual Counting Bottleneck in Vision-Language Models (ICML 2026 · similarity 0.92)
• Vision Language Models are Biased (ICLR 2026 · similarity 0.90)
• Language Models Need Inductive Biases to Count Inductively (ICLR 2025 · similarity 0.89)
• CountGD: Multi-Modal Open-World Counting (NeurIPS 2024 · similarity 0.88)
• UNICBench: UNIfied Counting Benchmark for MLLM (CVPR 2026 · similarity 0.87)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.87)