Research map / Label-efficient and robust learning / Self-supervised and contrastive
Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAE-CT, a sequential method that freezes a pre-trained MAE encoder, initializes an NNCLR head, then contrastively tunes the top half of the ViT to form object-centric clusters without labels, with minimal or standard augmentations.
Problem
Pre-trained MAE representations encode background as well as objects and lack abstraction, so they need large labeled data for downstream adaptation, while instance discrimination methods give object-focused features but rely on heavy expert augmentations and compute.
Approach
Use the NNCLR contrastive objective as a label-free "tuning" stage applied after MAE pre-training: first train an NNCLR head on the frozen encoder, then tune only the upper half of the ViT with layer-wise lr decay and a topk-NN lookup, imitating fine-tuning without labels.
Key results
- MAE-CT with ViT-H/16 achieves 82.2% linear probing accuracy on ImageNet (with augmentations), a new state-of-the-art among the compared methods trained on ImageNet only.
- MAE-CT_min with ViT-H/16 reaches 58.0% k-means cluster accuracy on ImageNet versus 11.1% for MAE.
- With ViT-L/16, MAE-CT_min (crop and flip only) surpasses MAE and iBOT on 1-shot (51.8%) and 2-shot (60.3%) low-shot evaluation, and matches or exceeds them on k-NN and linear probing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MAE-CT after MAE pre-training | MAE ViT-H/16 · ImageNet | k-means cluster accuracy: 11.1% to 58.0% | Helped |
| Apply MAE-CT with minimal augmentations | MAE ViT-H/16 · ImageNet linear probing | accuracy: 78.0 -> 81.5 | Helped |
| Apply MAE-CT with minimal augmentations | MAE ViT-H/16 · ImageNet 1 shot | accuracy: 9.0 -> 53.1 | Helped |
| Add NNCLR head initialization on frozen encoder features before contrastive tuning (ablation) | MAE-CT_min ViT-L/16 · ImageNet k-NN | accuracy: 68.1 -> 77.4 | Helped |
| Add nearest-neighbor lookup to NNCLR (ablation) | MAE-CT_min ViT-L/16 · ImageNet k-NN | accuracy: 40.5 -> 77.4 | Helped |
| Apply 75% masking during contrastive tuning (ablation) | MAE-CT_min ViT-L/16 · ImageNet k-NN | accuracy: 77.4 -> 75.3 | Hurt |
| Jointly pre-train MAE and NNCLR instead of sequentially applying contrastive tuning (ablation) | MAE ViT-L/16 · ImageNet k-NN | accuracy: 66.1 vs. 77.4 | Hurt |
| Reuse the NNCLR head from combined pre-training rather than reinitializing it for CT (ablation) | combined pre-training + CT ViT-L/16 · ImageNet k-NN | accuracy: 75.3 vs. 76.7 | Hurt |
| Increase topk-NN lookup k during contrastive tuning (ablation) | MAE-CT_min ViT-L/16 · ImageNet-R | accuracy: 41.2 -> 42.2 | Helped |
| Use extensive BYOL augmentations instead of minimal crop and flip | MAE-CT ViT-H/16 · ImageNet 1 shot | accuracy: 53.1 vs. 50.1 | Hurt |
| Add contrastive tuning after MAE pre-training | MAE ViT-H/16 on ImageNet-Dogs15 · ImageNet-Dogs15 | k-means cluster accuracy: 18.7% vs. 94.3% | Helped |
Surprising findings
- Combined MAE and NNCLR pre-training learns color-histogram shortcut features that contrastive tuning only partially removes, whereas the sequential approach avoids this shortcut.
- Topk-NN lookup with larger k improves performance during contrastive tuning, contrary to an ablation reported in prior work (NNCLR).
Limitations
- With ViT-B/16 the method does not reach the performance of the best ID methods in feature evaluation.
- Extensive augmentations remain superior for smaller models and for settings with more labels.
- Masking during contrastive tuning reduces performance, so the best results require no masking and more compute on the full encoder.
Experimental setup
Datasets: ImageNet-1K ImageNet-Dogs15
Benchmarks: ImageNet-1K ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A ImageNet-Dogs15
Metrics: top-1 accuracy linear probing accuracy k-NN accuracy low-shot accuracy k-means cluster accuracy silhouette score
Baselines: MAE MoCo v3 MSN iBOT Layer Grafting CMAE BootMAE I-JEPA
Builds on: Masked Autoencoders NNCLR SimCLR Layer Grafting partial fine-tuning
Closest papers by meaning
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.90)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.90)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.90)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.88)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.