Research map / Vision-language and multimodal / Multimodal LLMs
Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates state-of-the-art VLMs on all 100 original Bongard problems across open-ended, multiple-choice, per-image concept detection, and hypothesis generation tasks, comparing against a human study.
Problem
Whether modern vision-language models can solve Bongard problems, abstract visual puzzles requiring pattern recognition and rule induction, and how their performance compares to humans.
Approach
Diagnostic evaluation of VLMs on the original 100 Bongard problems using four complementary task settings (open-ended solving, multiple choice, per-image concept classification, hypothesis formulation), with an LLM judge and a new Prolific human study for comparison.
Key results
- o1 solves the most Bongard problems in the open-ended setting (43 of 100), ahead of Claude 3.5 Sonnet (31), while GPT-4o and Gemini 2.0 Flash solve 25 each.
- With 10 answer choices in the multiple-choice setting, o1 reaches 91 solved BPs, but the best human participant solved 68 BPs in the open-ended human study.
- o1 proposes correct hypotheses for 72 of 100 BPs while solving only 44 open-ended, and the LLM judge agreed with human annotators 91% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide all 100 rule pairs as multiple-choice options instead of requiring open-ended rule generation | Bongard problem solving · Bongard problems (100) | solved BPs: 43 -> 57 (o1); 31 -> 37 (Claude 3.5 Sonnet); 25 -> 23 (GPT-4o); 25 -> 27 (Gemini 2.0 Flash) | Mixed |
| reduce the multiple-choice options from 100 rule pairs to 10, while retaining the correct pair | Bongard problem solving · Bongard problems (100) | solved BPs: 57 -> 91 (o1); 23 -> 68 (GPT-4o); 37 -> 80 (Claude 3.5 Sonnet) | Helped |
| compare VLM solving with human solving | o1 and human participants · Bongard problems (100), open-ended | solved BPs: 68 BPs were solved (best human); 43 (o1) | Mixed |
| ask for hypotheses containing the ground-truth rules rather than directly solving the BP | o1 · Bongard problems (100) | correct hypotheses / solved BPs: 72 out of 100 BPs; 44 open question BPs | Helped |
| ask for hypotheses containing the ground-truth rules rather than directly solving the BP | LLaVA-OneVision · Bongard problems (100) | correct hypotheses / solved BPs: 32 correct hypotheses; 5 open question BPs | Helped |
| classify each image using provided ground-truth concepts instead of generating rules for the full puzzle | VLMs · Bongard problems, concept recognition (Task 2) | correct concepts: Gemini 2.0: 30 correct concepts | Mixed |
| evaluate spatial-reasoning Bongard problems | VLMs vs. top-performing humans · Spatial Bongard problems | solved BPs (%): under 25%; more than 60% | Hurt |
Surprising findings
- Models solve more BPs when given a set of candidate rules in some settings, yet elementary concepts such as left/right position (BP#8) remain unsolved in all attempts.
- Overlap between BPs solved in open-ended solving and BPs where per-image concept classification succeeded is small, and stronger models classify concepts relatively worse.
Limitations
- Bongard problems are a narrow, specialized benchmark that may not reflect real-world VLM use.
- Problems may appear in training data of the evaluated models.
- Reliance on an LLM judge introduces evaluation uncertainty, although agreement with humans was 91%.
Experimental setup
Datasets: Original 100 Bongard problems (high-resolution variant)
Benchmarks: Bongard problems (100 original)
Metrics: solved BPs (of 100) accuracy on multiple-choice per-image concept classification accuracy hypothesis success rate
Baselines: Human participants (Prolific study) Gemini 1.5 Pro Qwen2-VL-72B LLaVA-OneVision 72B InternVL 2.5 78B
Builds on: Bongard problems LLM-as-a-judge
Closest papers by meaning
- Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World (ICLR 2024 · similarity 0.93)
- Reasoning Limitations of Multimodal Large Language Models. A case study of Bongard Problems (ICML 2025 · similarity 0.91)
- MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.91)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.90)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
- PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.