Research map / Label-efficient and robust learning / Recognition and long-tail
Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives generalization bounds for supervised contrastive representation learning when tuples are built by recycling a fixed labeled pool (non-i.i.d.), via U-statistics and sub-sampled tuple empirical risk minimization.
Problem
Existing generalization analyses of contrastive representation learning assume i.i.d. input tuples, but in practice tuples are formed by reusing data points from a fixed labeled pool, which breaks tuple independence.
Approach
Model training as ERM over tuples drawn from a fixed labeled pool; estimate the population unsupervised risk with class-wise U-statistics and decouple them into blocks of i.i.d. tuples, then bound the gap between sub-sampled empirical risk and the U-statistic, yielding excess-risk bounds that depend on class counts and the number of negatives k.
Key results
- Sub-sampled empirical risk minimizer's performance approaches the full U-statistics minimizer as the number of sub-sampled tuples M grows (MNIST, n=100 and n=10000).
- Sub-sampled tuple training with recycled data outperforms fully independent tuples when labeled data is small (n=100).
- Sample complexity for the U-statistics minimizer displays linear relationships with the number of classes and number of negatives k in synthetic experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of subsampled tuples | Subsampled contrastive learning on MNIST · MNIST, n=100 | performance | Helped |
| Reuse labeled samples across tuples instead of using only independent tuples | Subsampled contrastive learning · MNIST, n=100 | performance | Helped |
| Increase the number of subsampled tuples | Subsampled contrastive learning · MNIST, n=100 | performance relative to all-tuples training | Helped |
| Increase the number of classes (ablation) | Supervised contrastive learning · Synthetic data, k=3 | sample complexity | Hurt |
| Increase the number of negative samples (ablation) | Supervised contrastive learning · Synthetic data, |C|=5 | sample complexity | Hurt |
| Train with supervised contrastive learning rather than direct cross-entropy | Shallow neural network · Experiments | performance | Helped |
Surprising findings
- Recycling data across tuples outperforms i.i.d. tuple training when labeled data is scarce.
Limitations
- Bounds depend on 1/sqrt(rho_min), which can be pessimistic with many small classes.
- Square-root dependence on number of classes when k is small, arising from independent class-wise concentration.
- Experiments are small-scale (MNIST and synthetic data) with shallow networks.
Experimental setup
Datasets: MNIST synthetic Gaussian data
Benchmarks: MNIST
Metrics: linear classifier accuracy sample complexity N_epsilon
Baselines: i.i.d. tuple CRL analysis (Awasthi et al. style) cross-entropy training
Builds on: N-pair loss U-statistics decoupling Rademacher complexity Dudley entropy integral
Closest papers by meaning
- A Refined Generalization Analysis for Extreme Multi-class Supervised Contrastive Representation Learning (ICML 2026 · similarity 0.93)
- Generalization Analysis for Deep Contrastive Representation Learning (AAAI 2025 · similarity 0.92)
- Optimal Sample Complexity of Contrastive Learning (ICLR 2024 · similarity 0.89)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics (NeurIPS 2025 · similarity 0.87)
- Towards Generalization beyond Pointwise Learning: A Unified Information-theoretic Perspective (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.