Research map / Vision-language and multimodal / Multimodal LLMs

Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Prompting

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Graph-of-Mark, a training-free pixel-level visual prompt that draws query-filtered scene graphs (objects as masks, spatial relations as labeled arrows) onto images to improve zero-shot spatial reasoning in open-source multimodal LMs.

Problem

Multimodal language models overlook spatial relationships between objects; existing visual prompting such as Set-of-Mark treats marked objects as isolated entities, missing relational structure needed for spatial reasoning in VQA and referring expression comprehension.

Approach

Overlay a depth-aware scene graph directly onto the input image (object masks with ID marks, directed arrows labeled with spatial relations), built automatically from open-vocabulary detectors, a segmenter, and a monocular depth estimator, and filtered by the text query; no fine-tuning is needed, and the graph is rendered as pixels rather than verbalized as text.

Key results

• On Gemma-3-4B, GoM with numeric IDs and relation labels reaches 63.2 on GQA and 74.2 on VQAv1 (vs 56.2 and 64.3 for raw image).
• On LlamaV-o1-11B, GoM reaches 83.6 on VQAv2 and 57.6 on RefCOCOg, the highest absolute scores reported.
• Paper claims GoM improves base accuracy in VQA and localization by up to 11 percentage points; average latency 1.13 s/image vs 0.92 s for SoM.

Surprising findings

• Visual scene-graph prompting outperforms verbalized textual scene graphs by up to about 10%.
• Qwen-2.5-VL is adversely sensitive to Set-of-Mark style overlays, frequently failing to reference correct regions.

Limitations

• Textual labels occasionally hurt REC performance, attributed to added noise.
• Benefit peaks at moderate graph size (3-10 entities, 4-16 relations); surplus annotations add noise.
• Relies on an ensemble of external detectors, segmenter, and depth model whose errors propagate into the graph.

Experimental setup

Datasets: GQA VQAv1 VQAv2 RefCOCOg

Benchmarks: GQA VQAv1 VQAv2 RefCOCOg

Metrics: accuracy IoU>=0.9 accuracy for REC mean and std across 27 decoding runs latency

Baselines: Set-of-Mark segmentation-only prompting raw image

Builds on: Set-of-Mark Weighted Boxes Fusion Segment Anything (SAM-HQ)

Closest papers by meaning

• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.91)
• See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.89)
• Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.89)
• ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.89)
• Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.89)
• Compositional Chain-of-Thought Prompting for Large Multimodal Models (CVPR 2024 · similarity 0.89)