Research map / Vision-language and multimodal / Multimodal LLMs
Beyond Counting: Evaluating Abstract and Emotional Reasoning in Vision-Language Models
Yuan Zhou, Yan Zhang, Jianlong Chang, Xin Gu, Ying Wang, Kun Ding, Guangwen Yang, Shiming Xiang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EmojiGrid, a diagnostic benchmark probing fine-grained, higher-order, abstract and emotional reasoning in vision-language models.
Problem
Existing VLM benchmarks focus on coarse-grained object recognition or simple relational reasoning, leaving fine-grained and higher-order (abstract, emotional) reasoning largely unexamined.
Approach
A diagnostic benchmark (EmojiGrid) built around emoji-based grid visual stimuli to test abstract and emotional reasoning in VLMs; details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from perception tasks to abstraction tasks | Gemini-2.5-Pro · EmojiGrid | accuracy (%): Perception 71.23 -> Abstraction 75.46 (Counting 60.04) | Hurt |
| scale up model size within a family (Qwen-VL2.5 7B to 72B) | Qwen-VL2.5 · EmojiGrid overall | accuracy (%): 38.62 -> 51.02 | Improved |
| scale up Gemma3 from 4B to 27B | Gemma3 · EmojiGrid overall | accuracy (%): 33.93 -> 40.82 | Improved |
| scale up Intern-VL3 from 8B to 78B | Intern-VL3 · EmojiGrid overall | accuracy (%): 44.14 -> 47.09 (14B 46.64, 38B 46.74) | Mixed |
| use reasoning-enhanced model instead of standard model (Gemini-2.5-Pro vs GPT-4o) | closed-source VLMs · EmojiGrid overall | accuracy (%): 49.98 -> 73.37 | Improved |
| reasoning-enhanced open model QVQ-72B-Preview versus non-reasoning Qwen-VL2.5-72B | 72B open VLMs · EmojiGrid overall | accuracy (%): 51.02 -> 34.83 | Hurt |
| longer reasoning output | reasoning VLMs · EmojiGrid-lite | accuracy vs token count | No clear effect |
| Chinese instead of English questions | most VLMs · EmojiGrid language split | accuracy | Hurt |
Experimental setup
Benchmarks: EmojiGrid
Closest papers by meaning
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.90)
- What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning (AAAI 2026 · similarity 0.89)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.89)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
