Research map / Vision-language and multimodal / Multimodal LLMs
Reasoning Limitations of Multimodal Large Language Models. A case study of Bongard Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates eight MLLMs (4 proprietary, 4 open-access) on Bongard Problems using multiple generation and binary-classification strategies, and introduces Bongard-RWR, a real-world-image version of synthetic BP concepts, finding weak abstract reasoning regardless of domain.
Problem
Whether multimodal LLMs can solve Bongard Problems, which require discovering abstract visual concepts from few examples and articulating them in natural language, and whether their weak synthetic-BP performance stems from domain specificity.
Approach
Designs seven prompting strategies (Direct, Descriptive, Contrastive and iterative/direct variants) for solving BPs with MLLMs, uses an MLLM voting ensemble to judge free-form answers, and builds Bongard-RWR, which re-expresses synthetic BP concepts with real-world photos so performance can be compared on the same concepts across domains.
Key results
- On synthetic BPs under Direct generation, the best model (GPT-4o) solved only 17 of 100 problems; Descriptive-direct with Claude 3.5 Sonnet or GPT-4o reached 22.
- On Bongard-RWR, the best generative result was 13 of 60 (Claude 3.5 Sonnet, Descriptive), while human participants averaged 39.2 of 60 (65%).
- In the Images to Sides binary task on OpenWorld, Gemini 1.5 Pro and Claude 3.5 Sonnet reached 96%, exceeding the 64% of the SNAIL baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Descriptive instead of Direct generation | MLLM answer generation · Synthetic BPs, HOI, OpenWorld, RWR | best correct answers: 17 to 21; 35 to 45; 40 to 57; 5 to 13 | Helped |
| Use Contrastive instead of Descriptive generation | MLLM answer generation · Synthetic BPs, HOI, OpenWorld, RWR | correct answers | Hurt |
| Add prior-turn dialog history to Descriptive reasoning (ablation) | Descriptive generation · All four BP datasets | correct answers | Hurt |
| Add prior-turn dialog history to Contrastive reasoning (ablation) | Contrastive generation · All four BP datasets | correct answers | Mixed |
| Add the whole-matrix image to Descriptive generation (ablation) | Descriptive generation · All datasets and models | correct answers: 12 out of 32 | Mixed |
| Add the whole-matrix image to Contrastive generation (ablation) | Contrastive generation · All datasets and models | correct answers | Helped |
| Use Contrastive-direct rather than Descriptive generation (ablation) | MLLM answer generation · Most datasets and models | correct answers | Hurt |
| Remove color from RWR images (ablation) | Phi-3.5V Images to Sides · RWR-G | problems solved: 1 -> 16; +15 | Helped |
| Crop RWR images to square format (ablation) | InternVL2 8B Images to Sides · RWR-S | problems solved: 16 -> 29; +13 | Helped |
| Crop RWR images to square format and remove color (ablation) | Phi-3.5V Images to Sides · RWR-SG | problems solved: 1 -> 18; +17 | Helped |
| Remove color or crop images (ablation) | MLLM binary classification: Ground-Truth and Incorrect Label · RWR variants | performance | No clear effect |
| Increase model size from InternVL2.5 8B to InternVL2.5 38B and 78B | InternVL2.5 concept selection · HOI, OpenWorld, RWR, Synthetic BPs | accuracy | Helped |
Surprising findings
- Descriptive (captioning each panel separately) outperformed direct whole-matrix prompting, and Contrastive strategies generally underperformed Descriptive ones.
- Models performed poorly on real-world Bongard-RWR despite real images, suggesting the difficulty lies in abstract concepts rather than the visual domain.
Limitations
- Evaluation limited to 100 instances per dataset due to computational cost.
- Human study used a small pool of 30 mostly academic participants.
- Open-access models run locally, and proprietary models are black boxes, complicating attribution of performance differences.
Experimental setup
Datasets: Bongard-HOI Bongard-OpenWorld synthetic Bongard problems (Bongard-LOGO)
Benchmarks: synthetic Bongard problems Bongard-HOI Bongard-OpenWorld Bongard-RWR
Metrics: number of solved problems accuracy
Baselines: Random classifier SNAIL Human participants
Builds on: Bongard-LOGO Bongard-OpenWorld Bongard-HOI
Closest papers by meaning
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.91)
- VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning (ICLR 2025 · similarity 0.91)
- Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad? (ICML 2025 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
- Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World (ICLR 2024 · similarity 0.91)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.