Research map / Vision-language and multimodal / Multimodal LLMs
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Bongard-OpenWorld, a real-world few-shot visual reasoning benchmark with open-vocabulary free-form concepts, and evaluates few-shot learners, VLM+LLM pipelines, and a neuro-symbolic method against humans (best 64% vs 91% human).
Problem
Few-shot visual reasoning over novel, compositional, real-world visual concepts is poorly covered: existing benchmarks use synthetic diagrams or closed-vocabulary, fixed-structure concepts, and current models lag far behind humans.
Approach
Bongard problems built from free-form open-vocabulary concepts mined from CC-3M via grid sampling plus crowd-sourced commonsense concepts, with real web images, distractors, and hard negatives; benchmarks few-shot learners, VLM+LLM pipelines, and a neuro-symbolic logic approach.
Key results
- SNAIL with OpenCLIP representation and auxiliary captioning achieves 64.0% average accuracy, the best learner, versus 91.0% for humans.
- GPT-4 with InstructBLIP captions reaches 63.8% average accuracy; GPT-4V with raw images reaches 54.0%.
- Zero-shot ChatGPT with InstructBLIP captions drops to 51.4%, while the neuro-symbolic approach reaches 55.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenCLIP pretraining instead of scratch initialization (ablation) | SNAIL · Bongard-OpenWorld test | accuracy: 49.8 -> 61.3 | Helped |
| Add the auxiliary captioning task (ablation) | SNAIL with OpenCLIP · Bongard-OpenWorld test | accuracy: 61.3 -> 64.0 (+2.7%) | Helped |
| Add the auxiliary captioning task (ablation) | Meta-Baseline with OpenCLIP · Bongard-OpenWorld test | accuracy: 55.3 -> 56.3 (+2.5%) | Mixed |
| Add the auxiliary captioning task (ablation) | ProtoNet with OpenCLIP · Bongard-OpenWorld test | accuracy: 58.3 -> 58.5 (+0.2%) | Helped |
| Replace BLIP-2 captions with InstructBLIP captions (ablation) | ChatGPT · Bongard-OpenWorld test | accuracy: 58.8 -> 51.4 | Hurt |
| Replace BLIP-2 captions with InstructBLIP captions (ablation) | GPT-4 · Bongard-OpenWorld test | accuracy: 61.6 -> 63.8 | Helped |
| Use ChatCaptioner multi-round captions instead of BLIP-2 captions (ablation) | ChatGPT · Bongard-OpenWorld test | accuracy: 58.8 -> 49.3 | Hurt |
| Fine-tune ChatGPT on Bongard-OpenWorld training data (ablation) | ChatGPT with BLIP-2 captions · Bongard-OpenWorld test | accuracy: 58.8 -> 63.3 | Helped |
| Increase visual pretraining vocabulary from ImageNet-1K to OpenCLIP (ablation) | ProtoNet · Bongard-HOI | accuracy: 58.3 -> 71.3 | Helped |
| Increase visual pretraining vocabulary from ImageNet-1K to OpenCLIP (ablation) | ProtoNet · Bongard-OpenWorld | accuracy: 56.0 -> 58.3 | Helped |
Surprising findings
- Fine-tuned ChatGPT with BLIP-2 captions outperforms zero-shot GPT-4 on explicit concept induction metrics by a large margin, while plain InstructBLIP captions hurt ChatGPT accuracy.
- A CLIP-based ProtoNet reaches 71.3% on Bongard-HOI but only 58.3% on Bongard-OpenWorld, showing open-vocabulary pretraining does not close the gap.
Limitations
- Best model remains about 27 points below human performance (64% vs 91%).
- Zero-shot VLM+LLM pipelines are limited by caption quality and distractors in captions.
- Multi-image reasoning VLMs (OpenFlamingo, Otter, GPT-4V) fail on the task.
Experimental setup
Datasets: CC-3M Bongard-OpenWorld
Benchmarks: Bongard-OpenWorld
Metrics: accuracy BLEU METEOR ROUGE-L CIDEr
Baselines: Meta-Baseline MetaOptNet ProtoNet SNAIL OpenFlamingo Otter ChatGPT GPT-4 GPT-4V ChatCaptioner
Builds on: Bongard Problems Bongard-HOI Bongard-LOGO CC-3M ChatCaptioner SNAIL
Closest papers by meaning
- Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad? (ICML 2025 · similarity 0.93)
- Reasoning Limitations of Multimodal Large Language Models. A case study of Bongard Problems (ICML 2025 · similarity 0.91)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.88)
- VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.88)
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale (ICML 2026 · similarity 0.88)
- OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.