Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Representation Learning from Arbitrary Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Arbitrary Self-supervised Learning (ASL), which combines MAE with patch-level feature-augmented dual-branch learning (L1 alignment) to avoid the conflict between global contrastive learning and MIM when pre-training on non-iconic or arbitrary-scene data.
Problem
Hybrid contrastive learning + masked image modeling methods perform well on ImageNet but generalize poorly to non-iconic, arbitrary-scene data because global-level contrastive consistency conflicts with MAE's patch-level objective.
Approach
Argues MAE is patch-level contrastive learning, and removes global-level constraints: creates dual-branch features via patch-level feature augmentation (dropout in linear embedding, or an extra Transformer layer at encoder end) and aligns decoder predictions patch-by-patch with L1 loss, without [cls] alignment.
Key results
- ASL pre-trained on COCO reaches 48.6% ImageNet linear probing and 50.3% COCO box AP with ViT-B, versus 46.8% / 49.3% for MAE.
- ASL on ImageNet-100 + COCO (4000 epochs) reaches 85.9% linear probing and 94.2% fine-tuning on ImageNet-100, versus 74.5% / 90.9% for MAE at 800 epochs.
- ASL on ImageNet-1K pre-training gives 69.2% LP vs 67.8% for MAE and 48.4% mIoU on ADE20K for the COCO+ImageNet-100 model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ASL instead of MAE for COCO pre-training | ViT-B · ImageNet-1K val after COCO pre-training | linear probing top-1 accuracy: 46.8% -> 48.6% | Helped |
| Use ASL instead of MAE for COCO pre-training | ViT-B · COCO val2017 | bbox mAP: 49.3% -> 50.3% | Helped |
| Train ASL for 4000 rather than 800 epochs | ASL · ImageNet-100 val; ImageNet-100 + COCO pre-training | linear probing top-1 accuracy: 79.6% -> 85.9% | Helped |
| Use ASL rather than MAE for ImageNet-100 + COCO pre-training | ViT-B · ImageNet-100 val | linear probing top-1 accuracy: 74.5% -> 79.6% | Helped |
| Use ASL rather than MAE for ImageNet-100 + COCO pre-training | ViT-B · ImageNet-100 val | fine-tuning top-1 accuracy: 90.9% -> 92.4% | Helped |
| Use ASL rather than MAE for pre-training | ViT-B · ADE20K val | mIoU: 48.1% -> 48.4% | Helped |
| Add dropout in the linear embedding without prediction alignment (ablation) | ASL feature augmentation · ImageNet-1K val after COCO pre-training | linear probing top-1 accuracy: 46.8% -> 46.5% | Hurt |
| Add prediction alignment to dropout in the linear embedding (ablation) | ASL feature augmentation · ImageNet-1K val after COCO pre-training | linear probing top-1 accuracy: 46.5% -> 48.5% | Helped |
| Add an asymmetric Transformer layer at the encoder end without prediction alignment (ablation) | ASL feature augmentation · ImageNet-1K val after COCO pre-training | linear probing top-1 accuracy: 46.8% -> 46.6% | Hurt |
| Add prediction alignment to the asymmetric Transformer layer (ablation) | ASL feature augmentation · ImageNet-1K val after COCO pre-training | linear probing top-1 accuracy: 46.6% -> 48.6% | Helped |
| Use L1 loss instead of cosine similarity loss for alignment (ablation) | ASL · COCO pre-training; ImageNet-1K linear evaluation | linear probing top-1 accuracy: +1.0% | Helped |
| Align the [cls] token in addition to patch predictions (ablation) | ASL · COCO pre-training; ImageNet-1K linear evaluation | linear probing top-1 accuracy: 48.6% -> 45.6% | Hurt |
Surprising findings
- Hybrid SOTA methods (CMAE, ccMIM, SIM) underperform MAE when pre-trained on COCO, suggesting global contrastive constraints hurt on non-iconic data.
- Adding [cls] global contrastive alignment on top of L1 loss degrades performance on COCO pre-training.
Limitations
- Gains on ImageNet-1K pre-training are modest (e.g., +1.4% LP) and the method is designed mainly for arbitrary/non-iconic data.
- Large-scale arbitrary-scene pre-training is only evaluated on COCO and ImageNet-100; no large web-scale experiments reported.
- Theoretical argument relies on approximations (uniform prior over patches, Monte Carlo estimate of the marginal).
Experimental setup
Datasets: MS COCO ImageNet-100 ImageNet-1K CIFAR-10 CIFAR-100 ADE20K
Benchmarks: ImageNet-1K linear probing ImageNet-100 linear probing and fine-tuning COCO object detection and instance segmentation (Mask R-CNN) ADE20K semantic segmentation CIFAR-10 CIFAR-100
Metrics: top-1 accuracy bbox AP mask AP mIoU
Baselines: MAE ccMIM CMAE SIM iBOT MoCo v3 BEiT CAE R-MAE DINO
Builds on: MAE SimCSE iBOT DINO ccMIM
Closest papers by meaning
- A Simple Framework for Contrastive Learning of Visual Representations (ICML 2020 · similarity 0.88)
- On the duality between contrastive and non-contrastive self-supervised learning (ICLR 2023 · similarity 0.88)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.87)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.87)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.87)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.