Research map / Vision-language and multimodal / Multimodal LLMs

Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates frontier reasoning models on ConceptARC across textual and visual inputs, with and without Python tools, and manually classifies the natural-language rules models give to see whether they capture intended abstractions or rely on shortcuts.

Problem

Accuracy on ARC-style benchmarks does not show whether AI models use humanlike abstractions or superficial shortcuts; the paper asks whether models that exceed humans on ARC-AGI-1 have the underlying abstract reasoning competence.

Approach

Pairs output-grid accuracy with manual classification of model-generated natural-language rules into correct-intended, correct-unintended, and incorrect, and compares textual versus visual modalities, reasoning effort, and Python tool access on ConceptARC.

Key results

• o3 with medium effort and textual input reached 77.1% pass@1 on ConceptARC, above the 73% human accuracy; in visual modality o3 reached only 5.6–29.2%.
• About 27% of o3's correct textual outputs rested on correct-unintended or incorrect rules, versus about 8% for humans' correct outputs.
• Across evaluated models and settings, rule-grid alignment exceeded 90% of tasks, and the best visual setting showed a visual error rate of about 49–77%.

Surprising findings

• Visual-modality models generated correct-intended rules more often than they produced correct output grids, suggesting perception and application failures rather than reasoning failures.
• Models frequently exploited integer color encodings as features, producing correct-unintended rules.

Limitations

• Only the ConceptARC dataset was studied; ARC tasks were not analyzed.
• Only pass@1 was reported, and high-effort reasoning settings and larger token budgets for Claude and Gemini were not tested.
• Manual rule classification involves subjectivity, and human rule data was incomplete.

Experimental setup

Datasets: ConceptARC unpublished human solution and rule data from Moskvichev et al. 2023

Benchmarks: ConceptARC

Metrics: pass@1 output-grid accuracy rule classification (correct-intended, correct-unintended, i rule-grid alignment visual error rate

Baselines: GPT-4o Llama 4 Scout Qwen 2.5 VL 72B human participants

Builds on: ConceptARC ARC-AGI o3-preview evaluation prompt

Closest papers by meaning

• Think Visually, Reason Textually: Vision-Language Synergy in Abstract Reasoning (CVPR 2026 · similarity 0.90)
• PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks (NeurIPS 2026 · similarity 0.90)
• Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective (ICML 2025 · similarity 0.89)
• MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.87)
• Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models (NeurIPS 2025 · similarity 0.87)