Research map / Architectures and efficiency / Attention and transformers
On the Intrinsic Limits of Transformer Image Embeddings in Non-Solvable Spatial Reasoning
Siyi Lyu, Quan Liu, Feng Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale ViT from Base to Huge (N=5) | ViT linear probe · LSA Level 3 (Non-Solvable) | Ratio loss at N=5: 0.7494 -> 0.7003 | Improved |
| scale ViT from Base to Huge (N=20) | ViT linear probe · LSA Level 3 (Non-Solvable) | Ratio loss at N=20: 0.8807 -> 0.8608 | No clear effect |
| scale DINOv2 from Base to Giant | DINOv2 linear probe · LSA Level 3 (Non-Solvable) | Ratio loss at N=5: 0.5976 -> 0.5474 | Improved |
| scale ResNet from R-50 to R-152 | ResNet linear probe · LSA Level 3 (Non-Solvable) | Ratio loss at N=20: 0.8386 -> 0.8269 | No clear effect |
| move from abelian translation (Level 1) to non-solvable 3D rotation (Level 3) | ViT-Base linear probe · LSA, N=20 | Ratio loss: 0.1456 -> 0.8807 | Hurt |
| move from solvable SE(2) (Level 2) to non-solvable SO(3) (Level 3) | DINOv2-Base linear probe · LSA, N=20 | Ratio loss: 0.3435 -> 0.7458 | Hurt |
| increase composition depth from N=5 to N=20 | ViT-Base linear probe · LSA Level 3 (Non-Solvable) | Ratio loss: 0.7494 -> 0.8807 | Hurt |
| increase composition depth from N=5 to N=20 | ViT-Base linear probe · LSA Level 1 (Abelian) | Ratio loss: 0.0567 -> 0.1456 | Hurt |
| use self-supervised DINOv2-Base instead of supervised ViT-Base | Frozen encoder · LSA Level 3 (Non-Solvable) | Ratio loss at N=20: 0.8807 -> 0.7458 | Improved |
| rotate out-of-plane (Rot Y 30 deg) instead of in-plane (Rot Z 30 deg) | DINOv2 OPA analysis · Atomic linearity (N=1) | OPA R2: 0.3054 -> -0.0286 | Hurt |
Closest papers by meaning
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.88)
- Vision Transformers provably learn spatial structure (NeurIPS 2022 · similarity 0.88)
- World Knowledge in the Weights: Reading Concept Circuits of Vision Transformers (ECCV 2026 · similarity 0.88)
- VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoning (ICLR 2026 · similarity 0.87)
- Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers (NeurIPS 2025 · similarity 0.86)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
