Research map / Vision-language and multimodal / Multimodal LLMs
Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LocVLM, a V-LLM instruction-tuned with textual image-coordinate objectives (location prediction, negative prediction, reverse-location prediction) plus pseudo-captioned COCO data, to improve spatial reasoning, VQA, hallucination reduction and region description.
Problem
Existing V-LLMs such as BLIP-2 and LLaVA have weak spatial reasoning and localization awareness, performing near random on simple left/right and above/below questions.
Approach
Encode image-space coordinates as plain text and train the LLM with three coordinate-based instruction fine-tuning objectives (location prediction, negative prediction, reverse-location prediction), with pseudo-captions from the V-LLM itself and SEEM pseudo-boxes for video, without extra location tokens or architectural heads.
Key results
- On the COCO-Spatial-27K left/right ICL setting, LocVLM-B reaches 76.5% overall accuracy versus 55.1% for LLaVA and 14.7% for BLIP-2.
- LocVLM-L achieves 63.5 on GQA and 78.2 on VQAv2 val, state of the art among compared methods trained without zero-shot setting.
- On region description with RefCOCOg val, LocVLM-B reaches METEOR 26.0 versus 13.5 for LLaVA without zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with image-space coordinate instruction fine-tuning | LocVLM-B · COCO-Spatial-27K, direct VQA | accuracy (All): 55.1 -> 69.5 | Helped |
| Train with image-space coordinate instruction fine-tuning | LocVLM-B · COCO-Spatial-27K, ICL VQA | accuracy (All): 55.1 -> 76.5 | Helped |
| Train with image-space coordinate instruction fine-tuning | LocVLM-L · GQA test-dev | accuracy (zero-shot): 48.7 -> 50.2 | Helped |
| Train with image-space coordinate instruction fine-tuning | LocVLM-Vid-B · ActivityNet-QA | accuracy (zero-shot): 35.2 -> 37.4 | Helped |
| Use integer-valued coordinate binning instead of normalized floating point coordinates (ablation) | LocVLM · GQA | accuracy: 46.1 -> 47.3 | Helped |
| Use integer-valued coordinate binning instead of deviation from image-grid anchors (ablation) | LocVLM · GQA | accuracy: 47.0 -> 47.3 | Helped |
| Add LocPred (ablation) | LocVLM-B · RefCOCOg test | METEOR: 10.3 -> 12.2 | Helped |
| Add NegPred (ablation) | LocVLM-B · RefCOCOg test | METEOR: 12.2 -> 12.5 | Helped |
| Add RevLoc (ablation) | LocVLM-B · RefCOCOg test | METEOR: 12.5 -> 20.7 | Helped |
| Use bounding-box locations instead of point locations with pseudo-data (ablation) | LocVLM-B · RefCOCOg test | METEOR: 20.6 -> 20.7 | No clear effect |
| Add object-description pseudo-data (ablation) | LocVLM-B · RefCOCOg test | METEOR: 11.6 -> 20.7 | Helped |
| Add video LocPred (ablation) | LocVLM-Vid-B+ · ActivityNet-QA | accuracy: 37.4 -> 37.6 | Helped |
Surprising findings
- LLaVA's training corpus has strong left/right accuracy disparity, which the authors attribute to keyword artifacts in LLaVA-Instruct-80K.
- Direct use of temporal locations together with spatial locations causes training collapse in the video variant.
Limitations
- Video variant fails to model temporal locations; combining temporal and spatial coordinates causes training collapse.
- Images downsampled to 224 or 336 resolution limit localization accuracy relative to detectors.
- Failure cases on crowded, cluttered, truncated or occluded objects.
Experimental setup
Datasets: COCO CC3M RefCOCO RefCOCO+ RefCOCOg GQA VQAv2 ActivityNet-QA MSRVTT-QA MSVD-QA TGIF-QA ADE-20K
Benchmarks: COCO-Spatial-27K GQA VQAv2 ActivityNet-QA MSRVTT-QA MSVD-QA TGIF-QA Hal-COCO Hal-ADE Hal-Act POPE RefCOCO
Metrics: accuracy METEOR precision recall F1
Baselines: BLIP-2 LLaVA LLaVA-v1.5 Shikra Kosmos-2 Video-ChatGPT POPE
Builds on: LLaVA Shikra CLIP LLaMA SEEM
Closest papers by meaning
- Expand VSR Benchmark for VLLM to Expertize in Spatial Rules (AAAI 2025 · similarity 0.90)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.90)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.89)
- Enhancing Spatial Reasoning Through Visual and Textual Thinking (AAAI 2026 · similarity 0.89)
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.