Research map / Vision-language and multimodal / Multimodal LLMs
Core Knowledge Deficits in Multi-Modal Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoreCognition, a 1,503-question MLLM benchmark over 12 developmental core-knowledge concepts, evaluates 230 models under 11 prompts, and proposes Concept Hacking, a controlled label-inversion test, to show MLLMs have core knowledge deficits and rely on shortcuts as they scale.
Problem
MLLMs excel at high-level perception and reasoning but fail on intuitive, low-level abilities that humans acquire in early childhood; it is unclear whether they possess core knowledge and whether scaling improves it.
Approach
Grounds a benchmark in developmental cognitive science (Piaget's stages) covering 12 core abilities, analyzes accuracy, cross-ability correlations, and scaling slopes, and adds Concept Hacking, which inverts labels by manipulating causal image features to distinguish genuine understanding from shortcuts.
Key results
- Best models GPT-o1 (74.91%), GPT-4o (69.25%), and Qwen2.5-VL-72B (68.29%) trail human average (86.98%) on CoreCognition.
- Low-level abilities (Sensorimotor and most Concrete Operational) show significantly smaller scaling slopes than Formal Operational abilities; perspective-taking declines with scale.
- Cognitive-instruction prompt raises average accuracy by over 6%, the best of 10 prompt variants; reasoning models show no significant average gain over instruction-tuned counterparts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate MLLMs on lower-level rather than higher-level core abilities | MLLMs · CoreCognition | accuracy: GPT-o1: 74.91\%, GPT-4o: 69.25\%, and Qwen2.5-VL: 68.29\%, trailing human performance by 15.91\%, 21.57\%, and 22.53\%, | Hurt |
| increase model size | MLLMs · CoreCognition, low-level abilities | scaling effect | Mixed |
| use reasoning-augmented models instead of instruction-tuned counterparts | MLLMs · CoreCognition | accuracy: P=0.0669; P=0.0037 | No clear effect |
| add a cognitive instruction prompt describing the task's conceptual target | MLLMs · CoreCognition | accuracy: over 6\% | Helped |
| use other prompt categories instead of cognitive instruction | MLLMs · CoreCognition | accuracy | No clear effect |
| manipulate task-relevant features to invert ground-truth labels | MLLMs · Concept Hacking; 45 paired CoreCognition samples | accuracy | Hurt |
| increase model size | MLLMs · Concept Hacking | core knowledge understanding | Hurt |
Surprising findings
- Reasoning (test-time scaling) models do not significantly improve core-knowledge abilities over instruction-tuned counterparts, and perspective-taking is worse for reasoning models.
- Larger models, including GPT-4o, fall into the shortcut-reliant quadrant under Concept Hacking, so size does not reduce susceptibility.
Limitations
- VQA format introduces auxiliary demands such as language, counting, and object recognition, and restricts testing to linguistic models.
- Concept Hacking is labor-intensive and does not scale to large benchmarks.
- Models still exploit shortcuts and spurious features in CoreCognition, potentially inflating scores.
Experimental setup
Datasets: CoreCognition
Benchmarks: CoreCognition SEED-Bench SEED-Bench-2 ChartQA
Metrics: accuracy chance-normalized accuracy Pearson correlation scaling slope control vs manipulation accuracy
Baselines: GPT-4o Qwen-VL-Max Gemini-1.5-Pro Claude-3.5-Sonnet GPT-o1 QVQ-72B-Preview InternVL3-78B
Builds on: Piaget's stage theory circular evaluation SEED-Bench
Closest papers by meaning
- Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study (NeurIPS 2025 · similarity 0.88)
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.87)
- DevBench: A multimodal developmental benchmark for language learning (NeurIPS 2024 · similarity 0.86)
- Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.86)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.86)
- Can Large Language Models Derive High-Level Cognition from Low-Level and Fragmented Foundational Information? (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.