Research map / Vision-language and multimodal / Visual question answering
COLUMBUS: Evaluating COgnitive Lateral Understanding Through Multiple-Choice reBUSes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COLUMBUS, a taxonomy-driven synthetic multiple-choice benchmark of over 1,000 text and icon rebus puzzles that tests visual lateral thinking in VLMs, finding a substantial gap to humans.
Problem
Visual question-answering benchmarks mainly test vertical thinking; lateral thinking in multimodal settings (rebus puzzles) is understudied for VLMs.
Approach
A three-step taxonomy-driven pipeline (18 latent rules, graph-based puzzle rendering with templates, and weighted Jaccard/Sentence-BERT distractor sampling) that automatically generates multiple-choice rebus puzzles from phrases and compounds.
Key results
- GPT-4o achieves 80.89% on COLUMBUS-text and 83.34% on COLUMBUS-icon, while humans score 98.00% and 93.21%.
- Adding human-curated node descriptions raises model accuracy by about 12-15 points (GPT-4o reaches over 90%), whereas relational-rule information adds only about 0.5-1.5 points.
- Backward chaining with belief graphs drops GPT-4o accuracy by about 14% on average across partitions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add forward chaining (ablation) | GPT-4o and Gemini models · COLUMBUS-text and COLUMBUS-icon | accuracy: -1.88%; +2.26% | Mixed |
| Add backward chaining (ablation) | GPT-4o · COLUMBUS-text and COLUMBUS-icon | accuracy: -14.1% | Hurt |
| Add backward chaining (ablation) | GPT-4o-mini · COLUMBUS-text and COLUMBUS-icon | accuracy: -22.86% | Hurt |
| Add a description of the puzzle type (ablation) | VLMs · COLUMBUS-text | accuracy: +2.68% | No clear effect |
| Add a description of the puzzle type (ablation) | VLMs · COLUMBUS-icon | accuracy: +3.39% | No clear effect |
| Add descriptions of the puzzle graph nodes (ablation) | VLMs · COLUMBUS-text | accuracy: +11.91% | Helped |
| Add descriptions of the puzzle graph nodes (ablation) | VLMs · COLUMBUS-icon | accuracy: +14.9% | Helped |
| Add descriptions of relational rules (ablation) | VLMs · COLUMBUS-text | accuracy: +1.47% | No clear effect |
| Add descriptions of relational rules (ablation) | VLMs · COLUMBUS-icon | accuracy: +0.56% | No clear effect |
| Combine individual rules with modifier rules | GPT-4o · COLUMBUS-text | accuracy: 3.02% | Helped |
| Use the taxonomy-based pipeline instead of DALLE-3 to generate rebuses | Rebus puzzle generation · 100 sampled puzzle answers | sound and complete visible elements: 98% vs 44% | Helped |
| Use the taxonomy-based pipeline instead of DALLE-3 to generate rebuses | Rebus puzzle generation · 100 sampled puzzle answers | human preference: 84% | Helped |
Surprising findings
- Backward chaining (belief graphs) substantially hurts performance of GPT-4o and GPT-4o-mini.
- DALL-E 3 produced sound and complete rebus element lists for only 44% of answers versus 98% for the taxonomy pipeline.
Limitations
- Benchmark scale is limited to 1,005 puzzles.
- Coverage of icon vs text puzzles and rules is imbalanced.
- Models struggle with rules that rearrange text, requiring puzzle-specific abstractions.
Experimental setup
Datasets: Wiktionary phrases theidioms.com LaDEC compound words emoji icons (Unicode 11.0)
Benchmarks: COLUMBUS-text COLUMBUS-icon
Metrics: accuracy
Baselines: CLIP BLIP-2 Fuyu-8b InstructBLIP Qwen-VL LLaVA CogVLM GPT-4o
Builds on: Sentence-BERT belief graphs BrainTeaser
Closest papers by meaning
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.90)
- Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.89)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.88)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.88)
- Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation (CVPR 2025 · similarity 0.88)
- MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.