Research map / Detection and segmentation / CNN features
Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-supervised ViT pre-training method that encodes position as a cropped vector field tied to contrastive crops and masks/predicts position embeddings alongside content, built on DINOv2, improving COCO detection and instance segmentation.
Problem
Self-supervised ViT features for object detection and instance segmentation need to capture both content and positional information, but conventional contrastive SSL applies the same position encodings to cropped views, discarding their position and scale within the original image.
Approach
Define positional encoding as a smooth vector field over the original image that is cropped with the same transformations as the views, plus random position/scale augmentation of the field, and apply masked prediction independently to both position and content embeddings within a DINOv2-style framework.
Key results
- With ViT-B/16 pre-trained on ImageNet-1K for 350 effective epochs, the method reaches 49.2 APBox and 43.8 APMask on COCO, above LOCA (48.3/43.0) and MAE (48.1/43.2).
- With ViT-S/16, the method reaches 44.8 APBox and 39.8 APMask on COCO, outperforming all compared methods except FLSL (45.5 APBox).
- Ablation with ViT-S: position encoding alone adds +1.9 APBox and position masked prediction alone adds +2.4 APBox over the 41.3 baseline; combined gives 44.8 APBox.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add crop-aligned position-embedding sampling (ablation) | Proposed method · COCO | APBox: 41.3 -> 43.2 | Helped |
| Add masked position prediction (Lpos) (ablation) | Proposed method · COCO | APBox: 41.3 -> 43.7 | Helped |
| Use both crop-aligned position sampling and Lpos (ablation) | Proposed method · COCO | APBox: 41.3 -> 44.8 | Helped |
| Sample scale s from Beta(2, 5) rather than use a constant scale (ablation) | Proposed method · COCO | APBox: 43.0 -> 44.8 | Helped |
| Sample scale s from Beta(2, 5) rather than uniformly from [0.2, 1.0] (ablation) | Proposed method · COCO | APBox: 43.7 -> 44.8 | Helped |
| Reduce position-field resolution from 50 × 50 to 19 × 19 (ablation) | Proposed method · COCO | APBox: 44.8 -> 44.4 | No clear effect |
| Use cross-wise rather than box-wise position-mask sampling (ablation) | Proposed method · COCO | APBox: 44.4 -> 44.8 | Helped |
| Use a 50:50 mix of content and position masking rather than content masking alone (ablation) | Proposed method · COCO | APBox: 43.4 -> 44.8 | Helped |
| Use a 50:50 mix of content and position masking rather than position masking alone (ablation) | Proposed method · COCO | APBox: 21.7 -> 44.8 | Helped |
| Use the proposed method rather than LOCA | ViT-Base/16 · COCO | APBox: 48.3 -> 49.2 | Helped |
| Use the proposed method rather than LOCA | ViT-Base/16 · COCO | APMask: 43.0 -> 43.8 | Helped |
| Use the proposed method rather than LOCA | ViT-Base/16 · ADE20K | mIoU: 48.5 -> 48.4 | No clear effect |
Surprising findings
- Applying position masking and prediction alone (100% of images) drops COCO APBox to 21.7, far below content-only masking, showing content masking is necessary.
- DINOv2 reproduced on ImageNet-1K only performs comparably to iBOT at about one-fourth the effective epochs.
Limitations
- Semantic segmentation on ADE20K shows no significant improvement over other top methods.
- Reproduction of baselines like FLSL and LOCA required the authors to identify bugs in official code; comparisons depend on reproduced or official weights.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K
Benchmarks: COCO object detection COCO instance segmentation ADE20K semantic segmentation
Metrics: APBox APMask mIoU
Baselines: DINO MAE iBOT DINOv2 Mugs DropPos CrIBo SelfPatch FLSL CrOC LOCA
Builds on: DINOv2 iBOT DINO ViTDet
Closest papers by meaning
- Patch-Level Representation Learning for Self-Supervised Vision Transformers (CVPR 2022 · similarity 0.88)
- Instance Localization for Self-Supervised Detection Pretraining (CVPR 2021 · similarity 0.88)
- Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation (CVPR 2023 · similarity 0.87)
- Efficient Self-supervised Vision Transformers for Representation Learning (ICLR 2022 · similarity 0.87)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.87)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.