Research map / Label-efficient and robust learning / Self-supervised and contrastive
Masking meets Supervision: A Strong Learning Alliance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskSub, a sub-branch training scheme that applies strong masking to a sub-network of the model and supervises it with a relaxed self-distillation loss from the unmasked main-branch, enabling stable supervised training with heavy masking augmentation.
Problem
Supervised learning struggles to use strong masking augmentation (mask ratios over 50%), since heavy masking degrades and destabilizes training, unlike masked image modeling pretraining.
Approach
Split training into a main-branch (standard unmasked loss) and a sub-branch (masked input) that shares weights with the main model; the sub-branch is supervised by the stop-gradient main-branch softmax output instead of hard labels, which relaxes the loss and stabilizes training. Variants DropSub and PathSub apply the same idea to dropout and drop-path.
Key results
- MaskSub on DeiT-III ViT-B/16 improves top-1 ImageNet accuracy from 83.5 to 84.1 at 400 epochs and 83.8 to 84.2 at 800 epochs.
- MaskSub on ResNet152 (RSB A2, 300 epochs) improves top-1 from 81.8 to 82.8 on ImageNet-1k.
- Applying MaskSub to BERT-base on GLUE raises average from 80.9 to 81.6, and CLIP ViT-B/32 zero-shot ImageNet accuracy rises from 33.5 to 37.6 with MaskSub, while plain masking drops it to 29.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MaskSub (50%) to DeiT-III training | ViT-B/16 · ImageNet-1k | Top-1 accuracy: 83.5 -> 84.1 (+0.6) | Helped |
| Add MaskSub (50%) to DeiT-III training | ViT-L/16 · ImageNet-1k | Top-1 accuracy: 84.5 -> 85.2 (+0.7) | Helped |
| Add MaskSub (50%) to MAE finetuning | ViT-B/16 · ImageNet-1k | Top-1 accuracy: 83.6 -> 83.9 (+0.3) | Helped |
| Add MaskSub (50%) to BEiTv2 finetuning | ViT-B/16 · ImageNet-1k | Top-1 accuracy: 85.5 -> 85.6 (+0.1) | Helped |
| Add MaskSub (50%) to CLIP finetuning | ViT-B/16 · ImageNet-1k | Top-1 accuracy: 84.8 -> 85.2 (+0.4) | Helped |
| Add MaskSub (50%) to ResNet-RSB A2 training | ResNet101 · ImageNet-1k | Top-1 accuracy: 81.4 -> 82.1 (+0.7) | Helped |
| Add MaskSub (50%) to Swin training | Swin-B · ImageNet-1k | Top-1 accuracy: 83.5 -> 83.9 (+0.4) | Helped |
| Apply masking augmentation to the main branch instead of using MaskSub | CLIP ViT-B/32 · ImageNet-1k | Zero-shot accuracy: 33.5 -> 29.8 (-3.7) | Hurt |
| Add MaskSub (50%) | CLIP ViT-B/32 · ImageNet-1k | Zero-shot accuracy: 33.5 -> 37.6 (+4.1) | Helped |
| Replace masked sub-branch with dropout (0.2) (ablation) | ViT-S/16 and ViT-B/16 with DeiT-III · ImageNet-1k | Top-1 accuracy: +0.2; +0.3 | Helped |
| Replace masked sub-branch with drop-path (baseline + 0.1) (ablation) | ViT-S/16 and ViT-B/16 with DeiT-III · ImageNet-1k | Top-1 accuracy: +0.4; +0.3 | Helped |
Surprising findings
- Plain masking at 50% ratio degrades CLIP zero-shot accuracy (33.5 to 29.8), while MaskSub improves it to 37.6.
- Plain masking on the main branch degrades standard train loss and accuracy, yet the masked sub-branch loss converges faster when combined with MaskSub.
Limitations
- Hierarchical architectures (ResNet, Swin) lack MAE-style token removal, so MaskSub doubles training cost there.
- Hyperparameters such as mask ratio are fixed at 50% without tuning per scenario, so optimal settings were not explored.
- Transfer gains are mixed; applying MaskSub during finetuning did not help on CIFAR datasets.
Experimental setup
Datasets: ImageNet-1k GLUE CIFAR-100 Flowers-102 Stanford Cars iNaturalist 2018 iNaturalist 2019 ADE-20k CIFAR-10
Benchmarks: ImageNet-1k GLUE ADE-20k CIFAR-100 Flowers-102 Stanford Cars iNaturalist 2018 iNaturalist 2019
Metrics: top-1 accuracy mIoU GLUE score zero-shot accuracy GPU days
Baselines: DeiT-III CoSub MAE SupMAE DataAugment GradAug ONE BYOT PS-KD SD-dropout MaskDistill BEiTv2
Builds on: DeiT-III masked autoencoder (MAE) CoSub RSB (ResNet strikes back) dropout drop-path
Closest papers by meaning
- MaskAnyNet: Rethinking Masked Image Regions as Valuable Information in Supervised Learning (AAAI 2026 · similarity 0.88)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.87)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.86)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.86)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.86)
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.