Research map / Vision-language and multimodal / Multimodal LLMs
Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coarse Correspondences, a training-free visual prompting method that uses an off-the-shelf tracker to overlay instance-ID markers on sparsely sampled frames, improving GPT-4V/4o spatial-temporal reasoning and LLaVA after fine-tuning.
Problem
MLLMs struggle with 3D spatial understanding and long-video temporal reasoning, and existing fixes rely on specialized architectures, 3D inputs, or task-specific fine-tuning.
Approach
Use a class-agnostic video tracker to find salient instance-level correspondences across frames or viewpoints, then draw the top-K tracked instance IDs as markers on a few sampled images so a general MLLM can reason about 3D space and time without architecture changes or task-specific training.
Key results
- On ScanQA, GPT-4o+CC improves CIDEr from 72.2 to 87.0 over GPT-4o with 4 frames.
- On EgoSchema, GPT-4o+CC reaches 73.2% accuracy with 8 frames versus 67.2% for GPT-4o.
- On R2R navigation (100 episodes), GPT-4o success rate rises from 12.00 to 23.00 with CC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Coarse Correspondences visual prompting | GPT-4O · ScanQA validation | BLEU-2: 19.8 -> 25.5 | Helped |
| Add Coarse Correspondences visual prompting | GPT-4O · ScanQA validation | METEOR: 14.8 -> 18.0 | Helped |
| Add Coarse Correspondences visual prompting | GPT-4O · OpenEQA EM-EQA | accuracy: 49.4 -> 59.1 | Helped |
| Add Coarse Correspondences visual prompting | GPT-4O · EgoSchema validation | accuracy: 67.2 -> 73.2 | Helped |
| Add Coarse Correspondences visual prompting | GPT-4O · R2R val-unseen | success rate: 12.00 -> 23.00 | Helped |
| Add Coarse Correspondences visual prompting | LLaVA · ScanQA validation | CIDEr: 67.3 -> 74.2 | Helped |
| Apply Coarse Correspondences during training only (ablation) | LLaVA · SQA3D | accuracy: 36.0 -> 37.17 | Helped |
| Also apply Coarse Correspondences during inference, in addition to training (ablation) | LLaVA · SQA3D | accuracy: 37.17 -> 39.13 | Helped |
| Add Coarse Correspondences visual prompting | GPT-4O · Camera motion diagnostic | harmonic mean accuracy: 53.9 -> 71.2 | Helped |
| Overlay all dense correspondences instead of retaining a handful of coarse correspondences (ablation) | Coarse Correspondences · ScanQA | performance | Hurt |
| Use segmentation masks instead of simple correspondence marks (ablation) | Coarse Correspondences · ScanQA | performance | Hurt |
| Add segmentation outlines to the correspondence marks (ablation) | Coarse Correspondences · ScanQA | object grounding | Helped |
Surprising findings
- Overlaying all dense correspondences degrades MLLM performance because marks occlude visual content.
- Applying CC only during fine-tuning (no CC at inference) still improves zero-shot SQA3D accuracy from 36.0 to 37.17.
Limitations
- Navigation evaluated on only 100 samples from val-unseen due to cost.
- EgoSchema evaluated on 500 validation questions only due to budget.
- Correspondence information is imprecise and coarse by design.
Experimental setup
Datasets: ScanQA OpenEQA EgoSchema R2R VLN-CE SQA3D
Benchmarks: ScanQA OpenEQA EM-EQA EgoSchema R2R (VLN-CE val-unseen subset) SQA3D
Metrics: BLEU-1 BLEU-2 METEOR ROUGE-L CIDEr accuracy success rate SPL navigation error harmonic mean accuracy
Baselines: ScanQA ScanRefer+MCAN 3D-LLM LongViviT MC-ViT-L LLoVi VideoAgent MVU LangRepo Set-of-Mark 3DAxiesPrompts Chain-of-thought
Builds on: Tracking Anything Set-of-Mark LLaVA NavGPT SAM 2
Closest papers by meaning
- From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models (ICML 2026 · similarity 0.92)
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.92)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching (CVPR 2026 · similarity 0.91)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.