Research map / Label-efficient and robust learning / Self-supervised and contrastive
On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NUCLR, a discriminative probabilistic model for self-supervised representation learning that uses multiple importance sampling and a non-parametric convex-optimization estimate of the partition-function popularities, yielding a new contrastive loss that beats SogCLR and CLIP on CC3M/CC12M zero-shot retrieval and classification.
Problem
Modeling the conditional density of a continuous-domain target (data prediction) for self-supervised representation learning requires computing an intractable partition function per anchor; InfoNCE/global contrastive loss uses a crude uniform approximation that leaves a non-diminishing estimation er
Approach
Estimate the MIS denominators (per-sample popularities sum_j p(y_j|x_j')) by solving a convex non-parametric problem over dual variables zeta, then train the encoder with the resulting discriminative X-risk using SogCLR-style moving averages and a non-uniform margin trick.
Key results
- On CC3M, NUCLR achieves mean 40.97 across five downstream tasks vs 39.86 for SogCLR and 35.38 for CLIP.
- On CC12M, NUCLR achieves mean 45.61 vs 44.73 for SogCLR and 40.40 for CLIP; MSCOCO R@1 34.36 vs 33.91 (SogCLR).
- On CC3M ImageNet-R, NUCLR reaches 43.82 vs 42.65 for SogCLR, with gains reported as larger on ImageNet-R than ImageNet1k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NUCLR instead of SogCLR | ResNet-50 + DistilBert pretrained on CC3M · MSCOCO | Recall@1: 28.54 ± 0.25 -> 29.55 ± 0.26 | Helped |
| Use NUCLR instead of SogCLR | ResNet-50 + DistilBert pretrained on CC3M · Flickr30k | Recall@1: 52.20 ± 0.64 -> 53.55 ± 0.22 | Helped |
| Use NUCLR instead of SogCLR | ResNet-50 + DistilBert pretrained on CC12M · MSCOCO | Recall@1: 33.91 ± 0.26 -> 34.36 ± 0.13 | Helped |
| Use NUCLR instead of SogCLR | ResNet-50 + DistilBert pretrained on CC12M · ImageNet1k | top-1 accuracy: 49.82 ± 0.14 -> 49.82 ± 0.23 | No clear effect |
| Learn per-example ζ instead of fixing ζ to ζ₀ (ablation) | NUCLR · Downstream retrieval and classification | performance | Helped |
| Reduce positive-pair push strength using ξ (ablation) | NUCLR · Downstream retrieval and classification | performance | Helped |
| Freeze ζ during the first 5 epochs (ablation) | NUCLR · Downstream retrieval and classification | performance | Helped |
Surprising findings
- Freezing the popularity variables zeta for the first 5 epochs improves performance, which the authors attribute to early model weights being far from the optimum.
Limitations
- Requires storing 2n additional floating-point numbers for the popularity variables.
- Learning a neural network to predict popularities is left open.
- Experiments limited to bimodal image-text pretraining on CC3M/CC12M with ResNet-50 and DistilBERT at 30 epochs.
Experimental setup
Datasets: CC3M CC12M
Benchmarks: MSCOCO Flickr30k CIFAR100 ImageNet1k ImageNet-R
Metrics: Recall@1 top-1 accuracy
Baselines: CLIP SigLIP DCL CyCLIP SogCLR
Builds on: SogCLR InfoNCE multiple importance sampling (balance heuristic) CLIP FCCO
Closest papers by meaning
- Probabilistic Self-supervised Representation Learning via Scoring Rules Minimization (ICLR 2024 · similarity 0.88)
- AUC-CL: A Batchsize-Robust Framework for Self-Supervised Contrastive Representation Learning (ICLR 2024 · similarity 0.87)
- The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning (ICML 2026 · similarity 0.87)
- Statistical Consistency and Generalization of Contrastive Representation Learning (ICML 2026 · similarity 0.87)
- Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective (NeurIPS 2022 · similarity 0.87)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.