Research map / Label-efficient and robust learning / Recognition and long-tail
Variational Supervised Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates supervised contrastive learning as variational inference over latent class variables, deriving a posterior-weighted ELBO loss (VarCon) with class-centroid matching and a confidence-adaptive temperature.
Problem
Supervised contrastive learning (SupCon) lacks a principled probabilistic account, requires large in-batch negatives and quadratic pairwise comparisons, and depends on strong tailored augmentations, which hurts generalization and efficiency.
Approach
Treats the class label as a latent variable and maximizes an ELBO over class posteriors, replacing pairwise comparisons with per-sample interactions against on-the-fly class centroids, plus a confidence-adaptive temperature tau_2 that softens per-sample label targets based on the model's current class probability.
Key results
- VarCon reaches 79.36% Top-1 on ImageNet with ResNet-50 vs. 78.72% for SupCon, and 78.29% on CIFAR-100 vs. 76.57% for SupCon.
- On ImageNet-100, VarCon achieves 86.34% Top-1 vs. 85.06% for SupCon and 80.83% for the best self-supervised method (Barlow Twins).
- In few-shot ImageNet with 100 samples per class, VarCon reaches 37.81% Top-1 vs. 36.57% for SupCon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SupCon with VarCon | ResNet-50 · CIFAR-10 | Top-1 accuracy: 95.51_{\pm0.06} -> 95.94_{\pm0.07} | Helped |
| Replace SupCon with VarCon | ResNet-50 · CIFAR-100 | Top-1 accuracy: 76.57_{\pm0.10} -> 78.29_{\pm0.08} | Helped |
| Replace SupCon with VarCon | ResNet-50 · ImageNet-100 | Top-1 accuracy: 85.06_{\pm0.07} -> 86.34_{\pm0.09} | Helped |
| Replace SupCon with VarCon | ResNet-50 · ImageNet | Top-1 accuracy: 78.72_{\pm0.06} -> 79.36_{\pm0.05} | Helped |
| Replace SupCon with VarCon | ResNet-50 · ImageNet, N=100 per class | Top-1 accuracy: 36.57 -> 37.81 | Helped |
| Replace SupCon with VarCon | ResNet-200 with StackedRandAugment · ImageNet | Top-1 accuracy: 81.43 -> 81.87 | Helped |
| Add confidence-adaptive temperature (epsilon=0.02) instead of fixed temperature (ablation) | VarCon · ImageNet embeddings | ARI: 0.627 -> 0.634 | Helped |
| Add confidence-adaptive temperature (epsilon=0.02) instead of fixed temperature (ablation) | VarCon · ImageNet, tau1=0.1 | Top-1 accuracy: 78.87\% -> 79.36\% | Helped |
| Increase adaptive-temperature epsilon from 0.02 to 0.08 (ablation) | VarCon · ImageNet, tau1=0.1 | Top-1 accuracy: 79.36\% -> 78.42\% | Hurt |
| Train VarCon for 200 rather than 350 epochs | VarCon · ImageNet | KNN Top-1 accuracy: 78.53\% -> 79.11\% | Helped |
| Train VarCon with batch size 2048 rather than SupCon with batch size 4096 | ImageNet · ImageNet | Top-1 accuracy: 78.7\% -> 79.26\% | Helped |
Surprising findings
- VarCon with a SimAugment-only pipeline (ResNet-101, 79.98%) is competitive with SupCon using StackedRandAugment (80.24%).
- Reported gains over SupCon are small on ImageNet (about 0.6 points) and CIFAR-10 (0.43 points), and VarCon is roughly on par with SupCon on transfer learning (84.94% vs. 84.88% mean).
Limitations
- Relies on labeled data, so not applicable to settings with scarce labels.
- Class-centroid computation adds overhead for datasets with many classes.
- Cross-domain shifts may require recalibrating the adaptive temperature mechanism.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet ImageNet-ReaL ImageNet-C Food Birdsnap SUN397 Cars Aircraft VOC2007
Metrics: Top-1 accuracy Top-5 accuracy KNN accuracy ARI NMI purity mAP mean per-class accuracy
Baselines: SimCLR MoCo V2 BYOL SwAV VicReg Barlow Twins Cross-Entropy SupCon
Builds on: SupCon InfoNCE variational autoencoder label smoothing X-CLR
Closest papers by meaning
- A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics (NeurIPS 2025 · similarity 0.88)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- Separating common from salient patterns with Contrastive Representation Learning (ICLR 2024 · similarity 0.88)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.88)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.88)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.