Research map / Vision-language and multimodal / Human and embodied modeling
One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes occluded image contrastive learning (OCL), which masks patches to create semantic views and contrasts [CLS] tokens of two disjoint visible-patch groups with a T-distributed spherical similarity, without augmentations, reconstruction, or auxiliary heads.
Problem
Visual self-supervised pre-training is inefficient: MIM spends capacity on pixel-level high-frequency reconstruction, while contrastive learning relies on hand-crafted augmentations, large batches, and auxiliary networks, and suffers from conceptual redundancy between views.
Approach
Use random patch masking (no hand-crafted augmentation) to split visible tokens of one image into two disjoint groups as positive views, contrast their [CLS] tokens across a mini-batch with a T-SP similarity and symmetric cross-entropy, and use no decoder or projection head.
Key results
- ViT-L/16 OCL reaches 85.8% top-1 fine-tuning accuracy on ImageNet-1K, and 77.9% linear probing.
- ViT-B/16 OCL reaches 74.2% linear probing and 83.4% fine-tuning, below MAE's 83.6% FT at 1600 epochs, but with 800 epochs.
- Ablation on overall masking ratio shows 0.4 is best for ViT-L/16 (LIN 77.9, FT 85.8) versus 0.2 (84.4 FT) and 0.8 (83.0 FT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase overall masked ratio from 0.2 to 0.4 (ablation) | OCL with ViT-L/16 · ImageNet-1K | fine-tuning top-1 accuracy: 84.4 -> 85.8 | Helped |
| Increase overall masked ratio from 0.4 to 0.8 (ablation) | OCL with ViT-L/16 · ImageNet-1K | fine-tuning top-1 accuracy: 85.8 -> 83.0 | Hurt |
| Increase overall masked ratio from 0.3 to 0.6 (ablation) | OCL with ViT-B/16 · ImageNet-1K | fine-tuning top-1 accuracy: 83.4 -> 82.8 | Hurt |
| Increase overall masked ratio from 0.6 to 0.8 (ablation) | OCL with ViT-B/16 · ImageNet-1K | fine-tuning top-1 accuracy: 82.8 -> 81.6 | Hurt |
| Increase effective batch size from 512 to 2,048 (ablation) | OCL with ViT-L/16 · ImageNet-1K | linear probing top-1 accuracy: 71.6 -> 77.9 | Helped |
| Add a 2-layer MLP head instead of no MLP head (ablation) | OCL with ViT-L/16 · ImageNet-1K | fine-tuning top-1 accuracy: 85.8 -> 85.6 | Hurt |
| Add a 3-layer MLP head instead of no MLP head (ablation) | OCL with ViT-L/16 · ImageNet-1K | fine-tuning top-1 accuracy: 85.8 -> 85.6 | Hurt |
| Increase T-SP kappa from 4 to 64 (ablation) | OCL with ViT-L/16 · ImageNet-1K | fine-tuning top-1 accuracy: 84.8 -> 85.8 | Helped |
| Increase T-SP kappa from 64 to 128 (ablation) | OCL with ViT-L/16 · ImageNet-1K | fine-tuning top-1 accuracy: 85.8 -> 85.5 | Hurt |
Surprising findings
- Removing the MLP projection head does not degrade performance; 2- and 3-layer heads slightly lower fine-tuning accuracy and increase pre-training time.
- Pre-training ViT-L/16 takes about 133 A100 hours on 4 GPUs according to the abstract, while the paper's ablation tables report 533 hours, an inconsistency.
Limitations
- Results lag behind some baselines (e.g., ViT-B/16 linear probing vs. DINO and iBOT; ViT-L/16 detection vs. MixMAE), and the method is not clearly superior overall.
- Large effective batch sizes (2,048 to 9,600) are required, and the maximum useful batch size was not determined due to compute limits.
- Pre-training is evaluated only on ImageNet-1K; the paper does not test larger datasets or billion-parameter models.
Experimental setup
Datasets: ImageNet-1K ImageNet-A ImageNet-R ImageNet-Sketch MSCOCO ADE20K
Benchmarks: ImageNet-1K MSCOCO ADE20K ImageNet-A ImageNet-R ImageNet-Sketch
Metrics: top-1 accuracy linear probing accuracy AP box AP mask mIoU
Baselines: MAE I-JEPA MoCo v3 BEiT CAE DINO SiameseIM ccMIM ConMIM iBOT MixMAE SimMIM
Builds on: MAE MoCo v3 CLIP T-SP (t-distributed spherical metric) ConMIM
Closest papers by meaning
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.85)
- TiMix: Text-Aware Image Mixing for Effective Vision-Language Pre-training (AAAI 2024 · similarity 0.85)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.85)
- Concept-Aware Batch Sampling Improves Language-Image Pretraining (CVPR 2026 · similarity 0.84)
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.84)
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.