Research map / Label-efficient and robust learning / Self-supervised and contrastive
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FOLK, a frequency-based SSL method that adaptively masks image frequencies (Com/RCom filters) and adds a self-distillation teacher branch on unmasked views, pre-training ViT and ResNet-50 on ImageNet-1K.
Problem
Frequency-masking SSL such as MFM uses fixed low/high-pass filters that make reconstruction too easy and only exposes the model to frequency-masked images, which hurts data efficiency in fine-tuning and few-shot learning.
Approach
Select masked frequencies adaptively per image via magnitude thresholds (Com retains top-magnitude frequencies, RCom removes them), and combine MFM frequency reconstruction with a DINO-style EMA teacher that sees the unmasked view, distilling its features into a student that sees the frequency-masked view.
Key results
- ViT-S/16 fine-tuned on ImageNet-1K reaches 82.1% top-1 with 800-epoch pre-training, and 81.6% with 300 epochs (vs. MFM 81.6% at 300 epochs ... MFM reported 81.6 for ViT-S and FOLK 81.6 at 300 epochs).
- With ViT-S/16 and 10% ImageNet labels (200 epochs), FOLK 800-epoch achieves 76.3% top-1 at BLR 2e-4 without warm-up, versus AttMask 69.8% and MFM 57.7%.
- ResNet-50 few-shot with 300-epoch pre-training: FOLK gets 71.1% (10% labels) and 46.2% (1% labels) vs MFM 53.9% and 22.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase FOLK pre-training from 300 to 800 epochs | FOLK with ViT-S/16 · ImageNet-1K | Top-1 accuracy: 81.6 -> 82.1 | Helped |
| Increase FOLK pre-training from 300 to 800 epochs | FOLK with ViT-B/16 · ImageNet-1K | Top-1 accuracy: 83.4 -> 84.0 | Helped |
| Use FOLK instead of MFM | ResNet-50 · ImageNet-1K, 10% labeled data | Top-1 accuracy: 53.9 -> 71.1 | Helped |
| Use FOLK instead of MFM | ResNet-50 · ImageNet-1K, 1% labeled data | Top-1 accuracy: 22.7 -> 46.2 | Helped |
| Use FOLK instead of MFM | ResNet-50 · ImageNet-1K, full-data fine-tuning | Top-1 accuracy: 80.1 -> 80.1 | No clear effect |
| Replace MFM's constant filters with Com/RCom filters (ablation) | MFM, ViT-S/16 · ImageNet-1K, 10% labeled data, BLR=2e-4, WUp=0 | Top-1 accuracy: 57.7 -> 66.3 | Helped |
| Use FOLK instead of MFM + Com/RCom | ViT-S/16 · ImageNet-1K, 10% labeled data, BLR=2e-4, WUp=0 | Top-1 accuracy: 66.3 -> 71.2 | Helped |
| Use multiple thresholds .005,.01,.05 instead of threshold .05 (ablation) | Com/RCom filters · CIFAR-10 | Top-1 accuracy: 98.8 -> 99.1 | Helped |
| Use multiple thresholds .005,.01,.05 instead of threshold .01 (ablation) | Com/RCom filters · CIFAR-100 | Top-1 accuracy: 90.7 -> 90.9 | Helped |
| Use thresholds .002,.01,.07 instead of .005,.01,.05 (ablation) | Com/RCom filters · ImageNet-1K | Top-1 accuracy: 81.6 -> 81.6 | No clear effect |
Surprising findings
- On full-data ResNet-50 fine-tuning, FOLK gives no significant gain over MFM (80.1 for both), with gains concentrated in few-shot settings.
Limitations
- Full-data gains over MFM are small or absent for CNNs (ResNet-50 at 80.1 for MFM and FOLK).
- Thresholds set S=[0.005,0.01,0.05] chosen empirically; the paper notes sensitivity only tested on a few sets.
- Robustness to noise evaluated on only 100 ImageNet validation images with a loss-based proxy.
Experimental setup
Datasets: ImageNet-1K ADE20K
Benchmarks: ImageNet-1K ADE20K CIFAR-10 CIFAR-100
Metrics: top-1 accuracy mIoU
Baselines: MAE iBOT BEiT AttMask MoCo V3 DINO MFM SimSiam MoCo v2 SimCLR BYOL SwAV
Builds on: Masked Frequency Modeling (MFM) DINO iBOT AttMask BEiT
Closest papers by meaning
- Frequency-Aware Transformer for Learned Image Compression (ICLR 2024 · similarity 0.86)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.86)
- One Meta-tuned Transformer is What You Need for Few-shot Learning (ICML 2024 · similarity 0.85)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.85)
- Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation (AAAI 2025 · similarity 0.85)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.