Research map / Vision-language and multimodal / Human and embodied modeling
Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Visual Riddles, a 400-item hand-curated benchmark of synthetic images with commonsense/world-knowledge riddles, hints and attributions, evaluated on open-ended VQA, multiple-choice VQA, and automatic judging; humans reach 82% versus at most 40% for the best VLM.
Problem
Existing visual commonsense and world-knowledge VQA benchmarks use pre-existing natural images that may have been seen in pretraining and limit scenario creativity; models lack integration of subtle visual cues with commonsense and world knowledge.
Approach
Designers create riddles by first devising a scenario with embedded visual clues and then generating a synthetic image (via text-to-image tools) and question, so images are novel and unseen in pretraining; adds hints, web attributions, multiple-choice and auto-rater tasks.
Key results
- Gemini-Pro-1.5 achieves 40% human-rated accuracy on open-ended riddles versus 82% for humans.
- On multiple-choice, GPT4o reaches 55% overall and 83% with hints; GPT-4 reaches 82% with attributions.
- Gemini-Pro-1.5 is the best auto-rater, matching human judgment 87% of the time in reference-based evaluation versus 52% reference-free.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a hint to the multiple-choice prompt | GPT4 on Visual Riddles · Visual Riddles multiple-choice VQA | accuracy: 45 -> 69 | Helped |
| Add an attribution to the multiple-choice prompt | GPT4 on Visual Riddles · Visual Riddles multiple-choice VQA | accuracy: 45 -> 82 | Helped |
| Add a hint to the multiple-choice prompt | GPT4o on Visual Riddles · Visual Riddles multiple-choice VQA | accuracy: 55 -> 83 | Helped |
| Add hints to open-ended questions | Gemini Pro 1.5 on Visual Riddles · Visual Riddles open-ended VQA | auto-rater rating accuracy: 53 -> 62 | Helped |
| Add attributions to open-ended questions | Gemini Pro 1.5 on Visual Riddles · Visual Riddles open-ended VQA | auto-rater rating accuracy: 53 -> 29 | Hurt |
| Provide human-generated oracle captions rather than model-generated captions (ablation) | Gemini Pro 1.5 · Visual Riddles open-ended VQA | human rating accuracy: 23 -> 50 | Helped |
| Exclude cannot-determine responses (ablation) | GPT4 on Visual Riddles · Visual Riddles multiple-choice VQA | accuracy: 45 -> 52 | Helped |
| Exclude cannot-determine responses (ablation) | Gemini Pro 1.5 on Visual Riddles · Visual Riddles multiple-choice VQA | accuracy: 38 -> 48 | Helped |
| Alter visual clues in the image while keeping the ground-truth answer (ablation) | Gemini Pro 1.5 answer validation · Visual Riddles modified-image answer validation | correctness rate: 74 -> 14 | Hurt |
| Alter visual clues in the image while keeping the ground-truth answer (ablation) | GPT4 answer validation · Visual Riddles modified-image answer validation | correctness rate: 68 -> 15 | Hurt |
| Give the auto-rater the ground-truth answer | Gemini Pro 1.5 judge · Visual Riddles automatic evaluation | accuracy compared to human rating: 52 -> 87 | Helped |
Surprising findings
- Only about 12-15% of generated images from open-source diffusion models matched the riddle prompts.
- Models frequently select the 'cannot determine' option, and accuracy rises substantially when hints or attributions are provided.
Limitations
- Small benchmark (400 riddles) intended for evaluation rather than training.
- Reference-free automatic evaluation correlates weakly with human judgment (52%).
- Some riddles may be culturally inappropriate or biased despite curation efforts.
Experimental setup
Datasets: Visual Riddles
Benchmarks: Visual Riddles
Metrics: accuracy human rating accuracy judge agreement with human rating
Baselines: Gemini-Pro-1.5 Gemini-Pro-Vision GPT-4 LLaVA-1.6-34B LLaVA-1.5-7B InstructBLIP-7B GPT4o Claude 3.5 Sonnet Qwen-VL-Max Molmo-7B
Builds on: VisIT-Bench Encyclopedic VQA OK-VQA WHOOPS!
Closest papers by meaning
- Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad? (ICML 2025 · similarity 0.87)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.87)
- PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.87)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.87)
- MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.