Research map / Architectures and efficiency / Attention and transformers
Can Transformers Capture Spatial Relations between Objects?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes precise physically grounded definitions for spatial relations, relabels SpatialSense into SpatialSense+, and introduces RelatiViT, an end-to-end ViT that masks subject/object regions and jointly models context and pair interaction, outperforming prior methods and bbox-only baselines.
Problem
Spatial relation prediction from images requires physically grounded relation labels; existing benchmarks (Rel3D synthetic, SpatialSense) have annotation inconsistencies, and prior vision methods fail to beat bbox-only baselines.
Approach
An end-to-end ViT that concatenates full-image patch tokens with masked subject and object image tokens into one sequence, so self-attention handles query localization, context aggregation and pair interaction together, rather than cascading a CNN or ViT with a separate relation head.
Key results
- RelatiViT reaches 80.09% avg. accuracy on Rel3D and 80.02% on SpatialSense+, versus 74.35% and 77.55% for the bbox-only baseline.
- GPT-4V achieves 68.26% avg. accuracy on SpatialSense+, the best of the VLMs tested, while MiniGPT-4 scores 50.00%.
- On the original SpatialSense, RelatiViT (65.79%) performs similarly to baselines, attributed to annotation language bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RelatiViT’s unified-sequence end-to-end design instead of RegionViT | ViT-based SRP · Rel3D | %Avg. Acc.: 72.43±0.55 -> 80.09±0.33 | Helped |
| use RelatiViT’s unified-sequence end-to-end design instead of RegionViT | ViT-based SRP · SpatialSense+ | %Avg. Acc.: 76.20±1.04 -> 80.02±0.73 | Helped |
| replace bbox-only prediction with RelatiViT visual features | spatial relation prediction · Rel3D | %Avg. Acc.: 74.35±0.77 -> 80.09±0.33 | Helped |
| replace bbox-only prediction with RelatiViT visual features | spatial relation prediction · SpatialSense+ | %Avg. Acc.: 77.55±0.78 -> 80.02±0.73 | Helped |
| add object-category embeddings to bbox-only inputs | bbox-only baseline · Rel3D | %Avg. Acc.: 74.35±0.77 -> 73.11±0.29 | Hurt |
| add object-category embeddings to bbox-only inputs | bbox-only baseline · SpatialSense+ | %Avg. Acc.: 77.55±0.78 -> 73.88±1.39 | Hurt |
| replace the CNN feature extractor with ViT | CNNTransformer · SpatialSense+ | %Avg. Acc.: 68.21±0.92 -> 72.42±1.28 | Helped |
| replace RoI Align with masked-image query localization | ViT + self-attention architecture · SpatialSense+ | %Avg. Acc.: 72.42±1.28 -> 74.40±2.14 | No clear effect |
| replace self-attention with cross-attention in the decoder | ViT + MaskImage architecture · SpatialSense+ | %Avg. Acc.: 74.40±2.14 -> 73.41±2.00 | No clear effect |
| remove pair interaction (ablation) | RelatiViT · Rel3D | %Avg. Acc.: 79.18±0.61 -> 80.09±0.33 | Hurt |
| remove pair interaction (ablation) | RelatiViT · SpatialSense+ | %Avg. Acc.: 76.37±0.86 -> 80.02±0.73 | Hurt |
| remove context aggregation, in addition to pair interaction (ablation) | RelatiViT · Rel3D | %Avg. Acc.: 70.08±0.41 -> 80.09±0.33 | Hurt |
Surprising findings
- Existing vision+bbox+class relation models (DRNet, VTransE, MotifsNet, RUNet) perform slightly worse than the bbox-only baseline, suggesting shortcut learning.
- Adding object category word embeddings (bbox-language) does not beat bbox-only on either dataset.
Limitations
- SpatialSense+ is small (about 5k training relation triplets, 9 predicates).
- VLM evaluation limited to SpatialSense+ and uses a fixed True/False prompt.
- Rel3D is synthetic with simple textures, limiting realism.
Experimental setup
Datasets: Rel3D SpatialSense+ SpatialSense
Benchmarks: Rel3D SpatialSense+ SpatialSense
Metrics: %Avg. Acc. %F1
Baselines: bbox-only bbox-language DRNet VTransE MotifsNet RUNet Vip-CNN PPR-FCN MiniGPT-4 LLaVA Gemini GPT-4V
Builds on: Vision Transformer IBOT DRNet SpatialSense Rel3D
Closest papers by meaning
- RelationVGGT : Visual Geometry Transformers for 3D Spatial Relation Segmentation (NeurIPS 2026 · similarity 0.89)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.87)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.86)
- RelationField: Relate Anything in Radiance Fields (CVPR 2025 · similarity 0.86)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.85)
- Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.