Research map / Vision-language and multimodal / Multimodal LLMs
Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stitch and Tell (SiTe), an annotation-free augmentation that stitches two image-caption pairs along a spatial axis and generates spatially-aware captions and yes/no spatial QA to improve VLM spatial understanding.
Problem
Vision-language models suffer from spatial hallucinations about relative object positions because image-text training data contains sparse spatial language relative to the rich spatial structure of images.
Approach
Stitch two images along a horizontal or vertical axis, fill spatial templates with their original captions, and derive spatial QA labels automatically from the known layout, so spatial supervision comes without annotation or large generator models.
Key results
- On LLaVA-v1.5-7B pretraining, SiTe-rand raises MME-Position from 127.83 to 133.33 (+5.50%) and Spatial-MM from 42.02 to 43.78 (+1.76%).
- With 5K spatial QA samples added in SFT, LLaVA SiTe-ratio reaches 48.58 on Spatial-MM (+6.56 points over baseline) and MME-Position 141.00 (+13.17).
- On CV-Bench 3D, LLaVA SiTe-ratio improves from 33.43 to 36.53 (+3.10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SiTe ratio-based stitching and spatial captions during pretraining | LLaVA-v1.5-7B on 558K · MME Position | score: +4.97 | Helped |
| Add SiTe ratio-based stitching and spatial captions during pretraining | LLaVA-v1.5-7B on 558K · Spatial-MM | accuracy (%): +2.13 | Helped |
| Add SiTe randomly paired stitching and spatial captions during pretraining | LLaVA-v1.5-7B on 558K · MME Position | score: +5.50 | Helped |
| Add SiTe ratio-based stitching and spatial captions during pretraining | LLaVA-Qwen2-1.5B on 558K · MME Position | score: +7.25 | Helped |
| Add 5K SiTe spatial QA samples per stitching direction during fine-tuning | LLaVA-v1.5-7B · Spatial-MM | accuracy (%): +6.56 | Helped |
| Add 5K SiTe spatial QA samples per stitching direction during fine-tuning | LLaVA-v1.5-7B · MME Position | score: +13.17 | Helped |
| Add 5K SiTe spatial QA samples per stitching direction during fine-tuning | LLaVA-Qwen2-1.5B · COCO-QA | accuracy (%): -1.86 | Hurt |
| Add SiTe spatially swapped contrastive examples | HALVA-7B · MME Position | score: -11.67 | Hurt |
| Add SiTe ratio-based stitching and spatial captions | LLaVA-v1.5-7B on 558K · CV-Bench 3D | accuracy (%): +3.10 | Helped |
| Use default stitch-to-raw ratio 1 : 3.58 rather than N=1K (ablation) | LLaVA-v1.5-7B on 558K · MME Position | score: 126.3 -> 133.33 | Helped |
| Use default stitch-to-raw ratio 1 : 3.58 rather than N=5K (ablation) | LLaVA-v1.5-7B on 558K · MME Position | score: 132.67 -> 133.33 | Helped |
| Use default stitch-to-raw ratio 1 : 3.58 rather than N=100K (ablation) | LLaVA-v1.5-7B on 558K · MME Position | score: 132.17 -> 133.33 | Helped |
Surprising findings
- Rotate and random-crop augmentations sometimes hurt spatial benchmarks when trained on Flickr30K, e.g., COCO-QA Spatial drops by 0.93% for Rotate.
- Adding spatial QA in SFT slightly reduces general benchmarks such as VQA-v2 and MM-Vet for LLaVA and COCO-QA for Qwen2-1.5B.
Limitations
- Only binary horizontal and vertical layouts are covered; diagonal, circular, depth, or 3D relations are not modeled.
- Gains are mixed in SFT stage, with drops on some general benchmarks for small models.
- Some reported HALVA results show declines on MME-Position and MMBench, and the HALVA contrastive setting is only partially described.
Experimental setup
Datasets: blip_laion_cc_sbu_558K Flickr30K llava_v1_5_mix665k Visual Spatial Reasoning (VSR)
Benchmarks: COCO-QA Spatial Spatial-MM MME-Position MM-Vet Spatial COCO-QA VQA-v2 MMBench MM-Vet CV-Bench RealWorldQA HR-Bench 8K V-Star
Metrics: accuracy percentage improvement over baseline
Baselines: Rotate augmentation Crop augmentation VSR data mixing HALVA
Builds on: LLaVA HALVA MixGen
Closest papers by meaning
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.90)
- SpatialCLIP: Learning 3D-aware Image Representations from Spatially Discriminative Language (CVPR 2025 · similarity 0.90)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.90)
- From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.