Research map / Optimization / Optimizers
Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SogCLR instead of SimCLR at batch size 128 | ResNet-50 contrastive pretraining · ImageNet-1K, 800 epochs | top-1 linear accuracy: 64.5 -> 67.9 | Helped |
| use SogCLR instead of SimCLR at batch size 256 | ResNet-50 contrastive pretraining · ImageNet-1K, 800 epochs | top-1 linear accuracy: 66.5 -> 69.4 | Helped |
| use SogCLR instead of SimCLR at batch size 512 | ResNet-50 contrastive pretraining · ImageNet-1K, 800 epochs | top-1 linear accuracy: 67.4 -> 69.6 | Helped |
| use SogCLR instead of SimCLR | ResNet-50 contrastive pretraining · ImageNet-S, 800 epochs, batch size 256 | top-1 linear accuracy: 77.4 -> 79.4 | Helped |
| use SogCLR instead of InfoNCE | CLIP-S bimodal training · ImageNet-S, 60 epochs, batch size 128 | accuracy: 63.4 -> 71.3 | Helped |
| use SogCLR instead of InfoNCE | CLIP-S bimodal training · ImageNet-S, 60 epochs, batch size 256 | accuracy: 64.9 -> 70.1 | Helped |
| add momentum averaging for u (gamma below 1) rather than setting gamma to 1 (ablation) | SogCLR · ImageNet-1K, batch size 256, 800 epochs | top-1 linear accuracy: 66.5 -> 69.4 | Helped |
| tune gamma within 0.6–0.99 (ablation) | SogCLR · ImageNet-1K, batch size 256, 800 epochs | top-1 linear accuracy: gamma 0.6: 69.2; gamma 0.7: 69.4; gamma 0.8: 69.3; gamma 0.9: 69.2; gamma 0.99: 69.2 | Mixed |
| add multi-crop views to a 4-layer projection head (ablation) | SogCLR · ImageNet-1K, batch size 256, 800 epochs | top-1 linear accuracy: 71.3 -> 72.5 | Helped |
| use a 4-layer rather than 3-layer projection head with multi-crop views (ablation) | SogCLR · ImageNet-1K, batch size 256, 800 epochs | top-1 linear accuracy: 71.7 -> 72.5 | Helped |
| use SogCLR rather than SimCLR with a wider ResNet-50 (2x) | ResNet-50 (2x) contrastive pretraining · ImageNet-1K, batch size 512 for SogCLR and 4096 for SimCLR | Top1: 74.2 -> 74.6 | Helped |
| use SogCLR rather than SimCLR with a wider ResNet-50 (4x) | ResNet-50 (4x) contrastive pretraining · ImageNet-1K, batch size 512 for SogCLR and 4096 for SimCLR | Top1: 76.5 -> 76.7 | Helped |
Closest papers by meaning
- Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective (NeurIPS 2022 · similarity 0.91)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.88)
- EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence (ICML 2024 · similarity 0.87)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- The Complexity of Finding Local Optima in Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.