Research map / Vision-language and multimodal / Multimodal LLMs

UNICBench: UNIfied Counting Benchmark for MLLM

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces UNICBench, a unified image/text/audio counting benchmark with a three-level capability taxonomy, difficulty tags, and a standardized evaluation toolkit, and benchmarks dozens of MLLMs on it.

Problem

MLLMs lack a unified, modality-spanning counting benchmark with standardized QA formats, ground truth, and evaluation protocols across image, text, and audio.

Approach

A unified QA-evidence schema covering image, text, and audio counting, with Pattern/Semantic/Reasoning capability levels, Easy/Medium/Hard difficulty tags from measurable attributes, deterministic numeric parsing, and tolerance-based hit rates for stratified reporting.

Key results

• On the image track, InternVL3.5-241B-A28B reaches 21.9% hit rate@100% and GPT-5-mini achieves the lowest overall MAE (29.8), while Qwen3-VL-30B-A3B-Thinking has very high overall MAE (1583.1).
• On the text track, Phi-4 has the lowest overall MAE (9.6) but only 69.7% success rate, while Gemini-2.5-Pro-Thinking reaches 63.3% hit rate@100%.
• On the audio track, GPT-Audio-mini achieves the lowest overall MAE (0.6), while Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B-Instruct reach 100% success with MAE around 29-30.

Surprising findings

• Models with lower success rates can show lower MAE because refused or unparseable outputs are excluded from error metrics, e.g., some GPT-Audio variants.
• Thinking mode does not uniformly help; some thinking variants have very large overall MAE on image counting (e.g., Qwen3-VL-30B-A3B-Thinking MAE 1583.1).

Limitations

• Image and audio counts are relatively small on audio and many image categories are drawn from existing datasets, limiting novelty of sources.
• Reported results contain internal inconsistencies (e.g., audio and text counts/category sizes, claimed 100% annotation consistency), and the paper's abstract/text numbers differ in places.
• Failure modes such as refusals and out-of-context outputs make MAE comparisons across models with different success rates non-comparable.

Experimental setup

Datasets: FSC-147 NWPU-MOC CARPK JHU-CROWD++ UCF-QNRF ShanghaiTech IOCfish5K Global Wheat Head Detection Snapshot Serengeti DESED AliMeeting

Benchmarks: UNICBench

Metrics: success rate MAE MSE hit rate@100% hit rate@90% hit rate@80%

Baselines: GPT-5 GPT-4o Gemini-2.5-Pro Qwen2.5-VL-72B Qwen2.5-Omni-7B Qwen3-Omni-30B-A3B DeepSeek-R1-0528

Builds on: FSC-147 NWPU-MOC DESED AliMeeting DocVQA

Closest papers by meaning

• Understanding Counting Mechanisms in Large Language and Vision-Language Models (CVPR 2026 · similarity 0.87)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.87)
• CountGD: Multi-Modal Open-World Counting (NeurIPS 2024 · similarity 0.87)
• SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.87)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.87)
• OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.86)