Research map / Label-efficient and robust learning / Self-supervised and contrastive
Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| graft MIM into lower layers and CL into higher layers instead of jointly optimizing both losses (ablation) | ViT-B/16 pre-training · ImageNet-1k, 1% few-shot | Top-1 accuracy: 47.6 -> 56.5 | Helped |
| reverse grafting order from MIM→CL to CL→MIM (ablation) | ViT-B/16 grafted pre-training · ImageNet-1k, 1% few-shot | Top-1 accuracy: 56.5 -> 32.5 | Hurt |
| jointly optimize MIM and CL losses on the same encoder (ablation) | ViT-B/16 pre-training · ImageNet-1k, 1% few-shot | Top-1 accuracy: 51.1 / 63.4 -> 47.6 | Hurt |
| use Layer Grafted Pre-training rather than MAE pre-training | ViT-B/16 representations · ImageNet-1k, linear evaluation | Top-1 accuracy: 68.0 -> 77.7 | Helped |
| use Layer Grafted Pre-training rather than Moco V3 | ViT-B/16 representations · ImageNet-1k, 1% few-shot | Top-1 accuracy: 63.4 -> 65.5 | Helped |
| replace higher-layer CL with VICReg in the grafted method (ablation) | ViT-B/16 grafted pre-training · ImageNet-1k, linear evaluation | Top-1 accuracy: 70.1% -> 74.9% | Helped |
| add l² regularization to preserve lower-layer MIM features (ablation) | ViT-L/16 Layer Grafted Pre-training · ImageNet-1k, linear evaluation | Top-1 accuracy: 80.5 -> 81.0 | Helped |
| increase the third-stage learning rate from 1.5e-6 to 1.5e-4 (ablation) | ViT-B/16 Layer Grafted Pre-training · ImageNet-1k, linear evaluation | Top-1 accuracy: 2.5%-63.0% -> 73.7%-75.1% | Helped |
| use a larger first-stage learning rate (1.5e-4 rather than 1.5e-6) (ablation) | ViT-B/16 Layer Grafted Pre-training with third-stage LR 1.5e-4 · ImageNet-1k, linear evaluation | Top-1 accuracy: 74.9%-75.1% -> 73.7%-73.8% | Hurt |
| scale Layer Grafted Pre-training from ViT-B/16 to ViT-L/16 | Layer Grafted Pre-training · ImageNet-1k, linear evaluation | Top-1 accuracy: 77.7 -> 81.0 | Helped |
Closest papers by meaning
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.91)
- MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Masked Image Modeling Representations (ICLR 2025 · similarity 0.90)
- Masked Image Modeling with Denoising Contrast (ICLR 2023 · similarity 0.90)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.89)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.89)
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.