Research map / Label-efficient and robust learning / Self-supervised and contrastive

Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax

ICLR 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Representation evaluation and trade-offs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adds independent noise to normalized encoder outputs inside the Deep InfoMax objective so that learned representations are provably pushed toward Gaussian or uniform distributions, with a modest accuracy trade-off.

Problem

Self-supervised representation learning methods like Deep InfoMax do not control the distribution of learned embeddings, while many downstream tasks (generative modeling, disentanglement, outlier detection) need embeddings matching a known prior such as Gaussian or uniform.

Approach

Inject independent Gaussian or uniform noise into the normalized encoder outputs while keeping the InfoMax objective; the decomposition of MI shows the objective upper-bounded by a maximum-entropy term, so maximizing it yields (asymptotically) exact Gaussian or uniform distribution matching, and the dual form connects it to contrastive losses with dot-product critics.

Key results

• On CIFAR-10 linear probing, SimCLR with noise sigma=0.1 reached 90.96% top-1 vs 90.83% baseline, and VICReg sigma=0.1 reached 68.92% top-1 on CIFAR-100 vs 65.71% baseline.
• On ImageNet-100 with VICReg, top-1 linear probing stayed within about 0.5 points across sigma in {0, 0.05, 0.1, 0.2} (72.18% at sigma=0 vs 71.68% at sigma=0.2).
• On ImageNet-1k with VICReg, top-1 linear probing was 67.41% at sigma=0 and 67.19% at sigma=0.2.

Measured effects reported in the paper

Surprising findings

• Noise injection slightly improved linear-probe accuracy on several CIFAR settings, contrary to the expectation of a clear trade-off.
• Uniform distribution matching is attained with equality only at a discrete grid, so the continuous uniform is only approached as the noise magnitude goes to zero.

Limitations

• On MNIST with low-dimensional (2D) embeddings, a moderate trade-off between noise magnitude and downstream classification quality is observed.
• Exact Gaussian distribution matching is achieved only asymptotically, and uniform matching only in the limit of vanishing noise.
• Generative experiments are limited to a small conditional GAN on MNIST.

Experimental setup

Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1k

Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1k MNIST

Metrics: top-1 accuracy top-5 accuracy p-value of normality tests (D'Agostino-Pearson, Shapiro-Wilk silhouette score ROC AUC

Baselines: SimCLR VICReg

Builds on: Deep InfoMax InfoNCE SimCLR VICReg Barlow Twins

Closest papers by meaning

• Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.85)
• Expressive Score-Based Priors for Distribution Matching with Geometry-Preserving Regularization (ICML 2025 · similarity 0.84)
• Dataset Distillation with Neural Characteristic Function: A Minmax Perspective (CVPR 2025 · similarity 0.84)
• Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.83)
• Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors (ICLR 2025 · similarity 0.83)
• Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.83)