Research map / Label-efficient and robust learning / Self-supervised and contrastive
Stochastic positional embeddings improve masked image modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StoP, which adds learned Gaussian noise to masked-token positional embeddings (with the noise projection tied to the context projection to prevent collapse) and applies it on top of I-JEPA to improve masked image modeling linear probing and downstream transfer.
Problem
Masked image modeling must predict semantic content at exact locations, but location is uncertain, so models overfit to location features; the paper aims to make MIM robust to location uncertainty.
Approach
Model masked-token positions as Gaussian random variables with a learned low-rank covariance (Sigma = sigma A A^T), sampled via reparameterization, and tie A to the context projection B so that the noise cannot collapse to zero without also removing context information.
Key results
- StoP on I-JEPA improves ImageNet linear probing with ViT-B/16 from 70.9% to 72.6% (last layer), and ViT-H/14 with 1% labels from 62.9% to 65.4%.
- StoP with ViT-H/14 reaches 79.0% top-1 linear probing on IN-1k, versus 78.2% for I-JEPA.
- On DAVIS 2017 video segmentation with ViT-L/16, StoP improves J&F from 55.9 to 58.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace sinusoidal positional embeddings with StoP | I-JEPA ViT-B/16 · ImageNet-1k linear probe, 100% labels, last layer | Top-1 accuracy: 70.9 -> 72.6 (+1.7%) | Helped |
| Replace sinusoidal positional embeddings with StoP | I-JEPA ViT-H/14 · ImageNet-1k linear probe, 1% labels, last layer | Top-1 accuracy: 62.9 -> 65.4 (+2.5%) | Helped |
| Replace sinusoidal positional embeddings with StoP | I-JEPA ViT-H/14 · iNat18 linear probe | accuracy: 47.6 -> 50.9 | Helped |
| Replace sinusoidal positional embeddings with StoP | I-JEPA ViT-L/16 · DAVIS 2017 semi-supervised segmentation | J&F Mean: +2.5% | Helped |
| Use learned deterministic positional embeddings instead of StoP (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | Top-1: 57.8 -> 54.4 | Hurt |
| Apply learned noise only to context tokens (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | Top-1: 57.8 -> 55.1 | Hurt |
| Apply learned noise to masked and context tokens (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | Top-1: 57.8 -> 56.8 | Hurt |
| Predict features at half resolution using bilinear resizing (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | Top-1: 57.8 -> 52.1 | Hurt |
| Predict features at half resolution using max pooling (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | Top-1: 57.8 -> 54.1 | Hurt |
| Learn the covariance matrix instead of using a fixed covariance matrix (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | accuracy: +3.5% | Helped |
| Use stochastic positions with σ=0.25 instead of deterministic positions (σ=0) (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | accuracy: 3.5% points | Helped |
| Add StoP instead of applying l1 regularization to A (ablation) | I-JEPA ViT-B · ImageNet-1k linear probe, 1% labels | accuracy: +2% | Helped |
Surprising findings
- Applying StoP to pixel-reconstruction MIM (MAE) was not successful, which the authors attribute to making pixel reconstruction too difficult.
- Predicting lower-resolution features via bilinear resizing or max pooling hurt performance (52.1 and 54.1 vs 54.3 baseline on the 1% linear probe).
Limitations
- Does not work with pixel-reconstruction MIM such as MAE; only applied to I-JEPA feature-space prediction.
- Models location uncertainty but not appearance uncertainty.
- Invariance-based methods such as iBOT and DINO still perform slightly better than MIM approaches.
Experimental setup
Datasets: ImageNet-1k
Benchmarks: ImageNet-1k linear probing ImageNet-1k 1% finetuning Places205 iNaturalist 2018 CIFAR100 CLEVR/Count CLEVR/Dist DAVIS 2017
Metrics: top-1 accuracy J&F mean
Baselines: I-JEPA MAE data2vec DINO iBOT MoCo v3 MSN SimCLR v2 BYOL
Builds on: I-JEPA Masked Autoencoders Vision Transformer
Closest papers by meaning
- Revealing the Dark Secrets of Masked Image Modeling (CVPR 2023 · similarity 0.86)
- Representation Learning by Detecting Incorrect Location Embeddings (AAAI 2023 · similarity 0.86)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.85)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.85)
- Pre-training with Random Orthogonal Projection Image Modeling (ICLR 2024 · similarity 0.84)
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.