Research map / Vision-language and multimodal / Human and embodied modeling
Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses mechanistic interpretability (attention-head analysis, DAS, linear interventions) to show fine-tuned CLIP and DINOv2 ViTs process same-different tasks in a perceptual stage followed by a relational stage.
Problem
How vision transformers compute abstract visual relations such as judging whether two objects are the same or different, given that ViTs often fail at relational tasks.
Approach
Applies mechanistic interpretability (local vs. global attention-head scoring, distributed alignment search, linear probing and interventions) to fine-tuned ViTs, revealing a two-stage algorithm: disentangled object representations formed locally, then relational comparison between object pairs.
Key results
- CLIP ViT-B/16 fine-tuned on discrimination and RMTS reaches 99.3% and 98.3% IID test accuracy respectively, while the from-scratch model gets chance on RMTS.
- CLIP ViT attains 93.9% zero-shot accuracy on the photorealistic dataset while other models are near chance (DINOv2 48%).
- Adding both disentanglement and pipeline losses lifts from-scratch RMTS test accuracy from 50.1% to 94.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add disentanglement loss (ablation) | From Scratch ViT-B/16 · Discrimination, IID test | accuracy: 76.5% -> 94.6% | Helped |
| Add pipeline loss (ablation) | From Scratch ViT-B/16 · Discrimination, IID test | accuracy: 76.5% -> 93.9% | Helped |
| Add disentanglement loss (ablation) | From Scratch ViT-B/16 · Discrimination, compositional test | accuracy: 76.0% -> 86.1% | Helped |
| Add pipeline loss (ablation) | From Scratch ViT-B/16 · Discrimination, compositional test | accuracy: 76.0% -> 92.3% | Helped |
| Add disentanglement loss (ablation) | From Scratch ViT-B/16 · RMTS, IID test | accuracy: 50.1% -> 54.4% | Helped |
| Add pipeline loss (ablation) | From Scratch ViT-B/16 · RMTS, IID test | accuracy: 50.1% -> 50.1% | No clear effect |
| Add both disentanglement and pipeline losses (ablation) | From Scratch ViT-B/16 · RMTS, IID test | accuracy: 50.1% -> 94.1% | Helped |
| Add both disentanglement and pipeline losses (ablation) | From Scratch ViT-B/16 · RMTS, compositional test | accuracy: 50.0% -> 77.4% | Helped |
| Fine-tune CLIP ViT-B/16 on the discrimination task | CLIP ViT-B/16 · Discrimination, photorealistic test | accuracy: 93.9% | Helped |
| Use DINOv2 ViT-B/16 instead of CLIP ViT-B/16 | Fine-tuned ViT on photorealistic discrimination · Discrimination, photorealistic test | accuracy: 93.9% -> 48% | Hurt |
| Use added or interpolated feature vectors rather than arbitrary sampled or random vectors | CLIP ViT-B/16 relational computation · Discrimination, novel-representation intervention | intervention success | Mixed |
| Use a from-scratch model rather than a pretrained ViT | ViT-B/16 on RMTS · RMTS, IID test | accuracy: 98.3% -> 50.1% | Hurt |
Surprising findings
- Models trained from scratch on RMTS reach only chance accuracy without auxiliary losses, and the disentanglement and pipeline losses must be combined to help.
- DINO and MAE, despite high in-distribution accuracy, show no two-stage processing and perform near chance on the photorealistic test set.
Limitations
- Case study on simple synthetic stimuli; the paper notes real tasks are likely far more complex.
- DAS may be overly expressive when misused, so interpretations rely on best practices.
- Two-stage pipeline appears in only some pretrained models, with pretraining scale as a speculative explanation.
Experimental setup
Datasets: synthetic same-different discrimination dataset synthetic RMTS dataset
Benchmarks: synthetic same-different discrimination test set RMTS test set photorealistic Blender same-different test set
Metrics: accuracy counterfactual intervention accuracy attention head score
Baselines: DINO DINOv2 ImageNet ViT MAE From Scratch ViT
Builds on: Distributed Alignment Search pyvene transformerlens
Closest papers by meaning
- Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning (ICLR 2025 · similarity 0.88)
- Disentangling and Integrating Relational and Sensory Information in Transformer Architectures (ICML 2025 · similarity 0.88)
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.88)
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.88)
- Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? (NeurIPS 2025 · similarity 0.88)
- Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.