Research map / Vision-language and multimodal / Multimodal LLMs
Expand VSR Benchmark for VLLM to Expertize in Spatial Rules
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-evaluates VLLMs on the VSR visual spatial reasoning benchmark, then expands training data with diffusion-based image repainting and text template rewriting plus a merged CLIP/SigLIP/DINOv2/SAM vision encoder to build a spatial expert (VSRE) built on LLaVA1.5.
Problem
VLLMs show inconsistent, instruction-sensitive, and vision-insensitive performance on visual spatial relation (VSR) questions, and there is a lack of unified evaluation and sufficient tuning data for positional reasoning.
Approach
Expand VSR triplets into large-scale instruction and image data (diffusion-based image-to-image, text-to-image, and inpainting repainting plus diverse question templates) and fuse four vision encoders (CLIP, SigLIP, DINOv2, SAM) into one merged visual encoder for LLaVA1.5.
Key results
- VSRE (LLaVA1.5 13B with merged 4-encoder vision and expanded data) reaches 79% on the VSR Test-S set versus about 52% for the LLaVA1.5 13B baseline, a reported 27% gain.
- On MME, VSRE scores 155.00 vs 133.33 for LLaVA1.5 13B; on MMBench 64.8 vs 57.6; on SEED-v2 46.6 vs 38.5.
- Scaling pre-training data from 100k to 400k raised LLaVA1.5 13B Test-G from 63.2 to 70.3 accuracy, with gains flattening beyond 400k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Expand training with turn-g 500k instead of no additional training (ablation) | LLaVA1.5 7B · VSR Test-G | accuracy: 54.3 -> 58.3 | Helped |
| Expand training with turn-g 500k instead of no additional training (ablation) | LLaVA1.5 13B · VSR Test-G | accuracy: 57.7 -> 62.5 | Helped |
| Increase pre-training data from pre-100k to pre-400k (ablation) | LLaVA1.5 13B with turn-g 500k · VSR Test-G | accuracy: 63.2 -> 70.3 | Helped |
| Increase pre-training data from pre-400k to pre-500k (ablation) | LLaVA1.5 13B with turn-g 500k · VSR Test-G | accuracy: 70.3 -> 70.2 | Hurt |
| Add turn-s 11k after pre-400k and turn-g 500k (ablation) | LLaVA1.5 13B · VSR Test-S | accuracy: 75.7 -> 76.6 | Helped |
| Use turn-s 11k rather than turn-g 11k for adapter pre-training (ablation) | LLaVA1.5 13B · VSR Test-G | accuracy: 59.2 -> 56.7 | Hurt |
| Use turn-s 11k rather than turn-g 11k for adapter pre-training (ablation) | LLaVA1.5 13B · VSR Test-S | accuracy: 68.2 -> 70.1 | Helped |
| Add pre-training data to turn-g 500k (ablation) | LLaVA1.5 7B · VSR Test-G | accuracy: 58.3 -> 66.7 | Helped |
| Add pre-training data to turn-g 500k (ablation) | LLaVA1.5 7B · VSR Test-S | accuracy: 69.5 -> 73.5 | Helped |
| Add pre-training data to turn-g 500k (ablation) | LLaVA1.5 13B · VSR Test-S | accuracy: 71.4 -> 75.7 | Helped |
| Combine SigLIP with DINOv2 (ablation) | LLaVA1.5 VSR model · VSR Test-G | accuracy | Helped |
| Add SAM rather than CLIP to SigLIP+DINOv2 (ablation) | LLaVA1.5 VSR model · VSR Test-G | accuracy | Helped |
Surprising findings
- Adding CLIP to an existing SigLIP+DINO+SAM combination gave less benefit than SAM on Test-G, attributed to feature overlap between CLIP and SigLIP.
- Tuning the adapter with only 11k IFT samples improved accuracy, but tuning the LLM on the same tiny 11k data made the model fail to respond.
Limitations
- Relies on the VSR dataset's triplets with images from MSCOCO, so the expansion is bounded by the seed data's spatial relation coverage.
- Evaluation is binary yes/no accuracy with single-token response parsing, which may not reflect open-ended spatial reasoning.
- Reported accuracies and comparison setups are not fully detailed in the main text; some table cells are missing (e.g., '-' entries).
Experimental setup
Datasets: VSR MSCOCO MME MMBench SEED-Bench
Benchmarks: VSR MME MMBench SEED-Bench v2
Metrics: accuracy normalized average intra-class distance
Baselines: LLaVA1.5 13B Qwen-VL BLIP2 InstructBLIP MiniGPT-4v2
Builds on: LLaVA-1.5 VSR dataset SDXL Cambrian-1 MoF
Closest papers by meaning
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.91)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.91)
- SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.