Research map / Image generation / Autoregressive and latent generation
Pre-training with Random Orthogonal Projection Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROPIM, a masked-image-modeling alternative that replaces binary patch masking with a count-sketch random orthogonal projection of patch embeddings and reconstructs via the complement projection, reaching 83.5% (ViT-B, 300 epochs) on ImageNet-1k.
Problem
Masked image modeling uses binary patch masking, which removes whole regions and yields a limited set of masking patterns; the paper seeks a corruption strategy that is continuous, with bounded noise, and cheap to pre-train.
Approach
Apply a count-sketch random orthogonal projection along the spatial (token) mode of patch embeddings, reconstruct raw pixels with an l1 loss on the complement of the projection (no MASK token, no dVAE tokenizer, no large decoder), so every token is softly corrupted with a bounded noise variance.
Key results
- ROPIM with ViT-B/16 reaches 83.5% top-1 on ImageNet-1k at 300 pre-training epochs and 84.0% at 800 epochs, above MAE's 83.1% at 800 epochs.
- On ADE20K with UperNet, ROPIM (ViT-B, 300 epochs) gets 48.5 mIoU versus 48.1 for MAE (1600 epochs).
- On ImageNet100 with ViT-T/16, ROPIM with l2 loss reaches 86.70% top-1 versus 85.08% for SimMIM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROPIM instead of MAE pre-training | ViT-B/16 · ImageNet-1k | top-1 accuracy: 83.1 -> 83.5 | Helped |
| Use ROPIM instead of MAE pre-training | ViT-B/16 · iNaturalist17 | top-1 accuracy: 70.5 -> 71.3 | Helped |
| Use ROPIM instead of MAE pre-training | ViT-B/16 · CIFAR10 | top-1 accuracy: 97.0 -> 97.6 | Helped |
| Use ROPIM instead of MAE pre-training | ViT-B/16 · CIFAR100 | top-1 accuracy: 82.5 -> 85.7 | Helped |
| Use ROPIM instead of MAE pre-training | ViT-B/16 · ADE20K | mIoU: 48.1 -> 48.5 | Helped |
| Use L2 reconstruction loss instead of L1 (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 86.43 -> 86.70 | Helped |
| Increase pre-training from 300 to 600 epochs (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 82.98 -> 84.20 | Helped |
| Increase pre-training from 600 to 800 epochs (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 84.20 -> 86.43 | Helped |
| Increase fine-tuning from 100 to 200 epochs (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 86.43 -> 87.80 | Helped |
| Increase fine-tuning from 200 to 300 epochs (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 87.80 -> 88.71 | Helped |
| Set sketching ratio rho to .14 instead of .07 (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 86.1 -> 86.4 | Helped |
| Set sketching ratio rho to .25 instead of .14 (ablation) | ROPIM with ViT-T/16 · ImageNet100 | top-1 accuracy: 86.4 -> 84.4 | Hurt |
Surprising findings
- The paper reports that binary masking removes crucial regions and degrades performance at high masking ratios, while ROP at sketching ratio 1/7 performs best, suggesting moderate lossy corruption of all tokens beats masking.
Limitations
- Gains over MAE at matched epochs are small (e.g., 83.5 vs 83.1 at 300 vs 800 epochs), and some comparisons use differing fine-tuning schedules.
- Baseline results on ImageNet100 were run by the authors with grid search, and BEiT there used a tokenizer trained on ImageNet-1k, so those comparisons are approximate.
Experimental setup
Datasets: ImageNet-1k ImageNet100 iNaturalist 2017 ADE20K CIFAR10 CIFAR100 Flowers102 CUB-200
Benchmarks: ImageNet-1k ImageNet100 iNaturalist 2017 CIFAR10 CIFAR100 ADE20K Flowers102 CUB-200
Metrics: top-1 accuracy mIoU total pre-training time
Baselines: DINO MoCo v3 BEiT MAE MFM CIM SimMIM LGP CAN data2vec
Builds on: masked autoencoders count sketch SimMIM BEiT
Closest papers by meaning
- Revealing the Dark Secrets of Masked Image Modeling (CVPR 2023 · similarity 0.89)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.88)
- Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN (ICML 2023 · similarity 0.88)
- Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation (AAAI 2025 · similarity 0.86)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.86)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.