Research map / Detection and segmentation / CNN features
Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply sparse, hierarchical masked pre-training rather than supervised pre-training | ConvNeXt-Base · ImageNet-1K val | top-1 accuracy: 83.8 -> 84.8 | Helped |
| Apply sparse, hierarchical masked pre-training rather than supervised pre-training | ConvNeXt-Base · COCO val2017 | APbb: +3.5 | Helped |
| Apply sparse, hierarchical masked pre-training rather than supervised pre-training | ConvNeXt-Base · COCO val2017 | APmk: 43.2 -> 45.1 | Helped |
| Apply SparK pre-training rather than contrastive pre-training | ResNet-50 · ImageNet-1K val | accuracy: 80.1 -> 80.6 | Helped |
| Replace sparse masking with zero-out masking (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 84.1 -> 83.2 | Hurt |
| Add sparse masking rather than zero-out masking (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 83.2 -> 84.1 | Helped |
| Remove hierarchical decoding (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 84.1 -> 83.6 | Hurt |
| Add hierarchical decoding (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 83.6 -> 84.1 | Helped |
| Add absolute positional embeddings (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 84.1 -> 83.9 | No clear effect |
| Calculate reconstruction loss on all positions instead of masked positions only (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 84.1 -> 83.3 | Hurt |
| Calculate reconstruction loss only on masked positions (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 83.3 -> 84.1 | Helped |
| Reduce pre-training from 1600 epochs to 800 epochs (ablation) | SparK on ConvNeXt-Small · ImageNet-1K val | top-1 accuracy: 84.1 -> 83.7 | Hurt |
Closest papers by meaning
- Revisiting Sparse Convolutional Model for Visual Recognition (NeurIPS 2022 · similarity 0.89)
- On Data Scaling in Masked Image Modeling (CVPR 2023 · similarity 0.89)
- MaskAnyNet: Rethinking Masked Image Regions as Valuable Information in Supervised Learning (AAAI 2026 · similarity 0.89)
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.88)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.88)
- DDAE: Towards Deep Dynamic Vision BERT Pretraining (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.