Research map / Vision-language and multimodal / Multimodal LLMs
See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes See&Trek, a training-free, GPU-free visual prompting method that selects semantically rich keyframes via YOLO detection and overlays VO-reconstructed camera-motion cues to boost MLLM spatial reasoning on VSI-Bench and STI-Bench.
Problem
Vision-only MLLMs struggle with spatial understanding from video because uniform frame sampling yields low-information frames and the model lacks explicit camera motion cues.
Approach
Select keyframes with the most detected object classes using a Balanced-Top-K strategy (object-rich, temporally diverse), estimate camera trajectory with ORB-based visual odometry, and overlay index/color markers on keyframes plus BEV and 3D trajectory plots as prompts, requiring no training.
Key results
- On VSI-Bench, See&Trek improves average accuracy for all tested open-source models, from +1.0% to +3.5% (largest on InternVL3-1B, 29.5 to 32.0).
- On STI-Bench, InternVL3-14B gains +1.4% average, with +8.6% on Trajectory Description.
- Ablation on InternVL3-8B: MSRS alone gives 41.8 and motion reconstruction alone 42.1 vs. 40.2 baseline; combined 43.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add See&Trek | InternVL3-1B · VSI-Bench | Avg.: 29.5 -> 32.0 (+3.5%) | Helped |
| Add See&Trek | InternVL3-14B · VSI-Bench | Avg.: 44.2 -> 45.6 (+1.4%) | Helped |
| Add See&Trek | Qwen2.5-VL-32B · VSI-Bench | Avg.: 33.7 -> 34.7 (+1.0%) | Helped |
| Add See&Trek | InternVL3-14B · STI-Bench | Avg.: 30.8 -> 32.2 (+1.4%) | Helped |
| Add See&Trek | Qwen2.5-VL-32B · STI-Bench | Avg.: 40.5 -> 41.7 (+1.2%) | Helped |
| Add Maximum Semantic Richness Sampling without Motion Reconstruction (ablation) | InternVL3-8B · VSI-Bench | Avg.: 40.2 -> 41.8 | Helped |
| Add Motion Reconstruction without Maximum Semantic Richness Sampling (ablation) | InternVL3-8B · VSI-Bench | Avg.: 40.2 -> 42.1 | Helped |
| Add both Maximum Semantic Richness Sampling and Motion Reconstruction (ablation) | InternVL3-8B · VSI-Bench | Avg.: 40.2 -> 43.2 | Helped |
| Add Spatiotemporal Encoding (ablation) | InternVL3-8B with neither encoding nor point prompts · VSI-Bench | Avg.: 42.4 -> 42.7 | Helped |
| Add point prompts (ablation) | InternVL3-8B with neither encoding nor point prompts · VSI-Bench | Avg.: 42.4 -> 42.7 | Helped |
| Set sampling interval N to 4 instead of N=1 (ablation) | InternVL3-8B + See&Trek · VSI-Bench | Avg.: 42.9 -> 43.2; 410 -> 82 | Mixed |
| Replace TopK with Balanced-TopK (ablation) | InternVL3-8B + MSRS · VSI-Bench | Avg.: 37.9 -> 43.2 | Helped |
Surprising findings
- Gains are small and sometimes negative on individual sub-tasks (e.g., Rel. Dir., Obj. Count), with larger models showing only modest improvements.
Limitations
- Gains are modest (typically 1-3%) and not uniform across sub-tasks.
- Processing time still depends on video length; VSI-Bench videos are over 1 minute.
- Relies on monocular VO with unknown scale and on an off-the-shelf detector's vocabulary.
Experimental setup
Datasets: VSI-Bench STI-Bench
Benchmarks: VSI-Bench STI-Bench
Metrics: accuracy Mean Relative Accuracy
Baselines: InternVL3-8B LLaVA-OneVision-7B LLaVA-NeXT-Video-7B Qwen2.5-VL-7B Kimi-VL-A3B GPT-4o Gemini-1.5 Pro
Builds on: ORB YOLO VSI-Bench
Closest papers by meaning
- Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.92)
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.91)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.91)
- Grounding Everything in Tokens for Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.