Research map / Vision-language and multimodal / Multimodal LLMs
What Makes a Maze Look Like a Maze?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deep Schema Grounding (DSG), an inference-only framework where an LLM extracts DAG schemas of abstract concepts and a VLM hierarchically grounds them on images to augment VQA, and introduces the 540-question Visual Abstractions Benchmark.
Problem
Vision-language models default to literal object interpretations and struggle to reason about visual abstractions (e.g., a maze built from tree branches) that are defined by lifted rules rather than concrete visual features.
Approach
Represent abstract concepts as schemas (dependency DAGs of primitive sub-concepts) generated by an LLM, hierarchically ground each schema component onto the image with a VLM, and feed the resolved schema back as context for question answering; the method is inference-only and model-agnostic.
Key results
- DSG with GPT-4o reaches 0.730 overall exact-match accuracy vs 0.664 for GPT-4o on VAB (+6.6 points).
- DSG improves counting questions by about 10 points (0.704 vs 0.604) for GPT-4o.
- DSG with LLaVA improves overall accuracy from 0.415 to 0.444.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hierarchical schema extraction, grounding, and schema context | GPT-4o · VAB | accuracy: 0.664 -> 0.730 | Helped |
| add hierarchical schema extraction, grounding, and schema context | GPT-4o · VAB Counting | accuracy: 0.604 -> 0.704 | Helped |
| add hierarchical schema extraction, grounding, and schema context | GPT-4o · VAB Strategic | accuracy: 0.653 ± 0.052 -> 0.751 | Helped |
| add hierarchical schema extraction, grounding, and schema context | GPT-4o · VAB Social | accuracy: 0.647 ± 0.163 -> 0.723 | Helped |
| add schema-guided grounding and QA context | LLaVA · VAB All | accuracy: 0.415 -> 0.444 | Helped |
| add schema-guided grounding and QA context | LLaVA · VAB Counting | accuracy: 0.344 -> 0.344 | No clear effect |
| add explicit grounding steps and multi-round execution (ablation) | GPT-4o with single-round schema components · VAB All | accuracy: 0.667 -> 0.730 | Helped |
| add hierarchical schema structure to single-round schema components (ablation) | GPT-4o · VAB All | accuracy: 0.667 -> 0.681 | Helped |
| add explicit hierarchical grounding instead of sequential grounding (ablation) | DSG · VAB All | accuracy: 0.693 -> 0.701 | Helped |
| add the full grounding history to final QA context (ablation) | DSG with hierarchical grounding · VAB All | accuracy: 0.701 -> 0.730 | Helped |
| use a 5-component rather than 3-component example schema prompt (ablation) | DSG · VAB All | accuracy: 0.730 -> 0.691 | Hurt |
| use a 7-component rather than 3-component example schema prompt (ablation) | DSG · VAB All | accuracy: 0.730 -> 0.685 | Hurt |
Surprising findings
- A language-only GPT-4o given the DSG grounded schema (no image) reaches 0.522, far above language-only baseline 0.397, showing the schema carries much of the signal.
- Generated schemas tend to be shorter than the number of components requested in the prompt, plateauing around 5-7 components.
Limitations
- VLMs struggle to ground spatially constrained schema components such as an unobstructed maze path.
- Schemas are unconstrained LLM outputs and may contain biases or ambiguous symbols.
- Benchmark is small (540 questions over 12 concepts) and the reported gains are on a single benchmark.
Experimental setup
Datasets: Visual Abstractions Benchmark
Benchmarks: Visual Abstractions Benchmark
Metrics: exact-match accuracy graded accuracy BERTScore LLM-based free-response accuracy
Baselines: ViperGPT VisProg LLaVA InstructBLIP GPT-4o GPT-4v Language-only GPT-4o
Builds on: ViperGPT VisProg chain-of-thought prompting theory-theory of concepts
Closest papers by meaning
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.89)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.89)
- Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad? (ICML 2025 · similarity 0.89)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.88)
- ARC Is a Vision Problem! (CVPR 2026 · similarity 0.88)
- CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.