Research map / Detection and segmentation / CNN features
DDAE: Towards Deep Dynamic Vision BERT Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deep Dynamic AutoEncoder (DDAE), a tokenizer-free masked image modeling framework with learnable, depth-specific dynamic loss weights based on patch distance to visible patches, combining pixel regression and feature self-distillation.
Problem
Existing masked image modeling methods reconstruct all masked patches equally, ignoring that reconstruction difficulty varies with distance from visible patches, which limits ViT representations (especially local inductive bias in deep layers).
Approach
Deep dynamic supervision: a learnable coefficient beta per supervised intermediate layer reweights masked-patch losses by their distance to visible patches (distance map), so the model shifts focus from easy to hard patches during pretraining; combined with EMA-based layer-wise feature self-distillation and pixel regression.
Key results
- DDAE ViT-B pretrained 100 epochs reaches 83.6% ImageNet top-1 fine-tuning, matching MAE 1600 epochs and surpassing BEiT 800 epochs.
- DDAE ViT-B at 800 epochs reaches 84.4% fine-tuning top-1 on ImageNet-1K, versus 83.4% for MAE 800 epochs.
- On ADE20K, DDAE ViT-B gets 49.6 mIoU vs 48.1 for MAE 1600 epochs; on COCO, 50.1 box AP vs 48.4 for MAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDAE instead of SdAE | ViT-B self-supervised pretraining · ImageNet-1K | fine-tuning accuracy: 84.0 -> 84.4 | Helped |
| Use DDAE instead of SdAE | ViT-B self-supervised pretraining · ADE20K | mIoU: 49.0 -> 49.6 | Helped |
| Use DDAE instead of SdAE | ViT-B self-supervised pretraining · COCO | box AP: 49.7 -> 50.1 | Helped |
| Use DDAE instead of SdAE | ViT-B self-supervised pretraining · COCO | mask AP: 43.3 -> 43.8 | Helped |
| Add DDS to pixel regression (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 83.0 -> 83.4 | Helped |
| Add DDS to feature self-distillation (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 83.0 -> 83.3 | Helped |
| Apply DDS to both pixel regression and feature self-distillation (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 83.0 -> 83.6 | Helped |
| Add deep supervision without dynamic loss (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 82.6 -> 82.8 | Helped |
| Add dynamic loss without deep supervision (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 82.6 -> 83.0 | Helped |
| Combine dynamic loss and deep supervision (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 82.6 -> 83.2 | Helped |
| Combine dynamic loss with feature self-distillation (ablation) | ViT-B MIM pretraining · ImageNet-1K | fine-tuning accuracy: 82.6 -> 83.3 | Helped |
| Increase supervisory number from K=4 to K=6 (ablation) | DDAE with ViT-S · ImageNet-1K | fine-tuning accuracy: 79.9 -> 79.9 | No clear effect |
Surprising findings
- Linear probing of DDAE (60.9% at 100 epochs, 68.1% at 800 epochs) lags contrastive methods like DINO and MoCo-v3, so the authors argue fine-tuning is a better metric for MIM.
Limitations
- Linear probing accuracy is weaker than contrastive methods.
- Pretraining cost per epoch is higher than MAE due to extra decoders and momentum encoder.
Experimental setup
Datasets: ImageNet-1K ADE20K COCO
Benchmarks: ImageNet-1K ADE20K COCO
Metrics: top-1 accuracy (fine-tuning) linear probing accuracy mIoU box AP mask AP
Baselines: MAE BEiT SimMIM MaskFeat DeepMIM SdAE DINO MoCo-v3 BEiT-v2
Builds on: masked autoencoder (MAE) BootMAE BEiT BEiT-v2 deep supervision
Closest papers by meaning
- Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling (ICLR 2023 · similarity 0.87)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.87)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.86)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.85)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.85)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.