Research map / Label-efficient and robust learning / Self-supervised and contrastive
Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WaMIM, which uses multi-level 2D discrete wavelet transform coefficients as multi-scale reconstruction targets for masked image modeling, with multi-layer decoders and loss weights, to speed up MIM pre-training on ImageNet-1K.
Problem
Pixel-based masked image modeling spends excessive compute reconstructing fine-grained textures from redundant raw pixels, prolonging pre-training; a compact, multi-scale reconstruction target is needed.
Approach
Perform 5-level Haar wavelet decomposition of the input image and assign coefficients of different levels as reconstruction targets for features from different encoder layers (shallow layers predict high-frequency coefficients, deep layers predict low-frequency ones), with adjustable per-target loss weights.
Key results
- WaMIM ViT-S reaches 82.0% top-1 on ImageNet-1K in 45 GPU hours (300 epochs) vs. MAE* 80.9% in 96 hours.
- WaMIM ViT-B reaches 83.8% top-1 in 96 GPU hours (400 epochs), 0.2% above MAE at 1600 epochs with about 13% of the compute.
- WaMIM Swin-B achieves 50.9 AP^b / 45.1 AP^m on COCO with Mask R-CNN, above LocalMIM (50.7/44.9) at 80% of its pre-training cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace pixel targets with wavelet targets | MIM with ViT-B · ImageNet-1K | Top-1 accuracy (%): 83.1 -> 83.8 | Helped |
| Replace pixel targets with wavelet targets | MIM with ViT-S · ImageNet-1K | Top-1 accuracy (%): +0.9% | Helped |
| Use WaMIM pre-training | Swin-B · COCO val2017 | APb: +0.2 | Helped |
| Use WaMIM pre-training | Swin-B · COCO val2017 | APm: +0.2 | Helped |
| Use WaMIM pre-training | ViT-B · ADE20K val | mIoU: 48.7 vs 48.8 | No clear effect |
| Increase mask ratio from 0.4 to 0.75 (ablation) | WaMIM ViT-B and Swin-B · ImageNet-1K | Top-1 accuracy (%): 83.1 -> 83.4; 83.6 -> 83.9 | Helped |
| Increase mask ratio from 0.75 to 0.9 (ablation) | WaMIM ViT-B and Swin-B · ImageNet-1K | Top-1 accuracy (%): 83.4 -> 82.9; 83.9 -> 83.4 | Hurt |
| Use 5 rather than 3 decomposition levels (ablation) | WaMIM ViT-B and Swin-B · ImageNet-1K | Top-1 accuracy (%): 83.2 -> 83.4; 83.7 -> 83.9 | Helped |
| Use a lightweight decoder (128D-4H) rather than 512D-16H (ablation) | WaMIM ViT-B and Swin-B · ImageNet-1K | Top-1 accuracy (%): 83.4 -> 83.2; 83.8 -> 83.9 | Mixed |
| Use uniformly spaced feature layers {3, 6, 9, 12} rather than {12} (ablation) | WaMIM ViT-B · ImageNet-1K | Top-1 accuracy (%): 82.9 -> 83.4 | Helped |
| Add multi-level features to layer {24} (ablation) | WaMIM Swin-B · ImageNet-1K | Top-1 accuracy (%): 83.2 -> 83.9 | Helped |
| Moderately emphasize low-frequency, high-level targets with weights {0.8,0.9,1.1,1.2} (ablation) | WaMIM ViT-B and Swin-B · ImageNet-1K | Top-1 accuracy (%): 83.3 -> 83.4; 83.8 -> 83.9 | Helped |
Surprising findings
- Loss weights that emphasize high-frequency, low-level targets (reverse ordering) reduce accuracy, and overemphasizing low-frequency targets also hurts performance, indicating a moderate balance is needed.
Limitations
- Efficiency is measured in GPU hours on a single A100, and some baselines are reproduced by the authors, so comparisons depend on the hardware setup.
- Loss weights and target/layer locations are hand-tuned via ablations on ViT-B and Swin-B; no automatic selection is proposed.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-C
Benchmarks: ImageNet-1K COCO ADE20K ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-C
Metrics: top-1 accuracy AP^b AP^m mIoU mCE GPU hours
Baselines: MAE SimMIM GreenMIM LocalMIM MFM MoCo v3 iBOT BEiT CAE MaskFeat
Builds on: MAE SimMIM GreenMIM LocalMIM MFM
Closest papers by meaning
- Masked Image Modeling With Local Multi-Scale Reconstruction (CVPR 2023 · similarity 0.90)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.89)
- Pre-training with Random Orthogonal Projection Image Modeling (ICLR 2024 · similarity 0.86)
- Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning (AAAI 2025 · similarity 0.86)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.86)
- WST: Wavelet-Based Multi-scale Tuning for Visual Transfer Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.