Research map / Vision-language and multimodal / Multimodal LLMs

PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PuzzleWorld, a benchmark of 667 multimodal puzzlehunt problems with final answers and human-annotated stepwise reasoning traces, and evaluates frontier and open multimodal LLMs, humans, and a fine-tuned InternVL3.

Problem

Existing reasoning benchmarks give well-specified tasks in constrained environments, so they do not test open-ended problem discovery from ambiguous multimodal clues. Puzzlehunts provide such a testbed, but no annotated benchmark exists for them.

Approach

Curates 667 open-ended puzzlehunt puzzles from Puzzled Pint with preserved original formats, plus human-written stepwise solution traces, skill and modality labels, and a stepwise LLM-judge metric that measures progress along the reference chain beyond final-answer accuracy.

Key results

• Gemini 3 Pro is the best model at 18.00% final answer accuracy and 39.99% stepwise accuracy, while human enthusiasts reach 44.44% accuracy and 51.70% stepwise.
• Fine-tuning InternVL3-8B on PuzzleWorld reasoning traces raises stepwise accuracy from 4.78% to 11.00%, while answer-only fine-tuning drops it to 2.96%.
• Most closed and open models score only 1-4% final answer accuracy; GPT-o3 gets stepwise 0 on most puzzles, and 53.33% of its bottleneck steps in 30 failures involved spatial or sketching skills.

Surprising findings

• Answer-only fine-tuning collapses stepwise accuracy below the base model (2.96% vs 4.78%).
• Several models score higher on structured puzzles than on unstructured visual ones, and QVQ-72B-Preview has higher stepwise accuracy (30.23%) than many closed models despite low final accuracy.

Limitations

• Benchmark uses only text and image puzzles; audio, video, interactive, and physical puzzles were excluded.
• Stepwise scoring relies on an LLM judge (GPT-4o) with only moderate validation (r=0.829 on 20 puzzles).
• Human baseline is small (9 novices, 9 enthusiasts), and expert results are assumed rather than measured.

Experimental setup

Datasets: PuzzleWorld Rebus puzzles MathVista

Benchmarks: PuzzleWorld Rebus puzzles MathVista

Metrics: final answer accuracy stepwise accuracy

Baselines: GPT-o3 GPT-4o Claude Opus 4 Gemini 2.5 Pro Gemini 3 Pro Grok 4 QVQ-72B-Preview InternVL3-78B Kimi VL A3B

Builds on: MMMU PuzzleVQA EnigmaEval

Closest papers by meaning

• PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.91)
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.89)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.89)
• HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games (ICLR 2026 · similarity 0.89)
• Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles (NeurIPS 2025 · similarity 0.88)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.88)