Research map / Label-efficient and robust learning / Self-supervised and contrastive
The Benefits of Balance: From Information Projections to Variance Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that data balancing (Sinkhorn-style alternating marginal rescaling, as used implicitly in CLIP and self-supervised clustering) reduces estimator variance, with a non-asymptotic MSE bound governed by spectral decay of the conditional-mean Markov operators.
Problem
Self-supervised and contrastive multimodal methods (CLIP, SwAV, DINO) implicitly apply data balancing to minibatch objectives, but it was unclear what statistical benefit this brings over empirical risk minimization on unbalanced minibatches.
Approach
Interprets balancing iterations as alternating information projections (KL, reverse KL, chi-squared) and linearizes them into alternating orthogonal projections via conditional-mean operators; the variance reduction per iteration is then exactly computable from the singular values of those operators, giving a non-asymptotic MSE bound with a constant-in-k first-order term.
Key results
- Theorem gives MSE bound of (sigma_0^2 - sigma_gap^2)/n + O(s^k/n) + O~(k^6/n^{3/2}), i.e. a strict first-order variance reduction over the empirical estimator.
- Balancing reduces variance by a quantity determined by singular values of the conditional-mean operators; the reduction is strictly positive whenever mu_X h is non-constant.
- In CLIP-type zero-shot classification on CIFAR-10, CIFAR-100 and STL-10, one balancing iteration already improves over the standard CLIP loss, and more iterations improve further.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| balance the data for one iteration | CLIP-type models · CIFAR-10, CIFAR-100, and STL-10 zero-shot classification | average per-class recall | Helped |
| use multiple balancing iterations instead of one (ablation) | CLIP-type models · CIFAR-10, CIFAR-100, and STL-10 zero-shot classification | average per-class recall | Helped |
| balance the metadata-curated training set | CLIP-type embedding model · ImageNet-Captions curation; smaller batch regime (m = 512) | zero-shot classification performance | Helped |
| balance the empirical measure to match the known target marginals | balanced estimator versus empirical-measure estimator · Statistical estimation under the paper's stated conditions | variance | Helped |
Surprising findings
- Balancing, originally proposed for other purposes, reduces the variance of the empirical estimate.
- The first-order variance reduction is constant in the number of iterations k, improving on prior bounds that grow exponentially in k.
Limitations
- Analysis assumes finite sample spaces and positive target marginals.
- CLIP experiments are small-scale with frozen base encoders and trainable heads.
- Second-order term is O(k^6 n^{-3/2}), so k must be scaled sublinearly in n for the asymptotic guarantee.
Experimental setup
Datasets: ImageNet-Captions LAION-2B
Benchmarks: CIFAR-10 CIFAR-100 STL-10
Metrics: average per-class recall mean squared error
Baselines: empirical measure estimator standard CLIP loss MetaCLIP
Builds on: CLIP SwAV Sinkhorn-Knopp matrix scaling
Closest papers by meaning
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.84)
- Mind the Graph When Balancing Data for Fairness or Robustness (NeurIPS 2024 · similarity 0.84)
- Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizes (NeurIPS 2025 · similarity 0.83)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.83)
- Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems (NeurIPS 2024 · similarity 0.83)
- Rethinking the Bias of Foundation Model under Long-tailed Distribution (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.