Research map / Vision-language and multimodal / Human and embodied modeling
Probing the Mid-level Vision Capabilities of Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 22 ImageNet-1K-pretrained self-supervised models on eight mid-level vision tasks (object segmentation, depth, surface normals, correspondence, image similarity) and relates them to ImageNet linear probing.
Problem
Self-supervised learning is mainly evaluated on high-level recognition tasks, so the mid-level vision capabilities (generic object localization, 3D geometric understanding, cross-view correspondence) of SSL models remain unexamined.
Approach
A controlled probing suite over SSL checkpoints trained on ImageNet-1K, using frozen features with a DPT decoder or training-free matching, to measure how mid-level performance correlates with high-level ImageNet linear probing and which factors (objective, architecture, size) drive mid-level performance.
Key results
- Mid-level performance correlates positively with ImageNet linear probing, but the correlation is weaker than among high-level tasks; generic object segmentation has R^2 = 0.70 while scene surface normals have R^2 = 0.08.
- MAE underperforms most SSLs on mid-level tasks despite competitive high-level performance, while MaskFeat outperforms most SSLs on mid-level tasks given similar high-level results.
- iBOT consistently leads most mid-level tasks, and ViT backbones generally outperform ResNet-50; ViT-L improves over ViT-B for iBOT and MAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ResNet-50 with ViT-B/16 in MoCo-v3 (ablation) | MoCo-v3 · Mid-level vision tasks | task scores: 71.48 -> 72.45; 27.22 -> 31.72; 43.03 -> 49.64 | Mixed |
| increase iBOT backbone from ViT-B/16 to ViT-L/16 (ablation) | iBOT · Mid-level vision tasks | task scores: 75.74 -> 75.32; 32.75 -> 35.65; 52.02 -> 54.53; 50.02 -> 52.56; 81.32 -> 85.14; 3.12 -> 3.16; 20.04 -> 14.79; 89.27 -> 8 | Mixed |
| increase MAE backbone from ViT-B/16 to ViT-L/16 (ablation) | MAE · Mid-level vision tasks | task scores: 65.77 -> 66.58; 22.67 -> 26.60; 43.89 -> 45.60; 26.78 -> 26.81; 66.17 -> 70.22; 2.74 -> 2.93; 6.64 -> 10.05; 82.91 -> 84 | Mixed |
| use HOG features as the masked prediction target instead of raw pixels or tokens | MaskFeat · Mid-level vision tasks | task performance | Helped |
| use contrastive learning objectives | contrastive SSL models · Depth and surface normal estimation | task performance | Mixed |
Surprising findings
- For several contrastive methods, higher ImageNet linear probing accuracy correlates with lower mid-level performance (negative relationship).
- CLIP performs much worse than most ImageNet-pretrained SSLs on mid-level tasks, and SAM shows weak ImageNet classification.
Limitations
- Study limited by compute resources, so it does not scale to more tasks and models.
- Excludes downstream tasks such as navigation and robotic manipulation.
- Restricted to SSL checkpoints publicly available and pretrained on ImageNet-1K, limiting generality to large-scale pretraining.
Experimental setup
Datasets: ImageNet-1K VOC07 VOC12 COCO NYUv2 NAVI ScanNet NIGHTS
Benchmarks: VOC07 VOC12 NYUv2 NAVI ScanNet NIGHTS ImageNet-1K
Metrics: mIoU F1 RMSE recall at thresholds 2D recall voting accuracy top-1 accuracy coefficient of determination R^2
Baselines: DINO iBOT MAE MaskFeat MoCo v3 CLIP CroCo SAM DINOv2
Builds on: Probe3D DPT AdaBins NIGHTS
Closest papers by meaning
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.87)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.86)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.86)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.86)
- Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding (CVPR 2025 · similarity 0.86)
- Localizing Memorization in SSL Vision Encoders (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.