Research map / Label-efficient and robust learning / Self-supervised and contrastive
Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention
Weichen Zhou, Yawen Zou, Chunzhi Gu, Ran Dong, Haoran Xie, Chao Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add point-wise MLP to the linear probe | DINOv2-L · NYU Depth V2 (frozen ViT-L, multi-layer features) | SA-delta1: 0.9157 -> 0.9325 | Improved |
| add global DPT decoder over the MLP probe | DINOv2-L · NYU Depth V2 (frozen ViT-L, multi-layer features) | SA-delta1: 0.9325 -> 0.9483 | Improved |
| add point-wise MLP to the linear probe | MAE-L · NYU Depth V2 (frozen ViT-L, multi-layer features) | SA-delta1: 0.6033 -> 0.6390 | Improved |
| add global DPT decoder over the linear probe | MAE-L · NYU Depth V2 (frozen ViT-L, multi-layer features) | SA-delta1: 0.6033 -> 0.7022 | Improved |
| add DPT decoder over the linear probe | iBOT-L · NYU Depth V2 (frozen ViT-L, multi-layer features) | SA-delta1: 0.8198 -> 0.8524 | Improved |
| restrict probe to random or residual control subspaces instead of the top-k task-aligned subspace (ablation) | linear probe · NYU Depth V2 | SA-delta1: collapses below 0.18 | Hurt |
| increase subspace rank to k=32 (ablation) | MAE-L linear probe · NYU Depth V2 | recovery of linear baseline: >98% of linear potential at k=32 | Improved |
| increase subspace rank to k>=64 (ablation) | DINOv2-L linear probe · NYU Depth V2 | recovery of linear baseline: converges at k >= 64 | Improved |
| extract layer-24 final readout instead of intermediate layers | DINOv2-L · NYU Depth V2 | geometric energy contribution: 35.8 / 36.7 at l12/l18 -> 10.3 at l24 | Hurt |
| choose layer 18, 21 or 24 as readout for normals, depth, segmentation | DINOv2-L · NYUv2 linear probes | peak layer by task: normals peak layer 18, depth layer 21, segmentation layer 24 | Mixed |
| use low rank k<=16 subspace across random probe seeds (ablation) | DINOv2-L · NYU Depth V2 | subspace similarity: >0.93 at k <= 16 | Improved |
Closest papers by meaning
- Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis (NeurIPS 2026 · similarity 0.88)
- GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR 2024 · similarity 0.86)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.86)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
- Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces (NeurIPS 2026 · similarity 0.85)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
