Research map / Label-efficient and robust learning / Self-supervised and contrastive

Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression

ICLR 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervision in supervised pipelines · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies self-supervised covariance estimation for deep heteroscedastic regression, proposing a non-commutative upper bound on the 2-Wasserstein distance and a neighborhood-based Mahalanobis pseudo-label for covariances.

Problem

Deep heteroscedastic regression must learn a sample-dependent covariance without ground-truth labels; unsupervised NLL-based methods trade off computational cost against accuracy and suffer from residual-driven optimization instability.

Approach

Derive an upper bound on the 2-Wasserstein distance between normals with non-commutative covariances, ||mu1-mu2||^2 + ||Sigma1^{1/2}-Sigma2^{1/2}||_F^2, which avoids eigendecomposition and is stable to optimize; pair it with pseudo-covariance labels computed from Mahalanobis-weighted kNN target variance.

Key results

• On UCI regression, 2-W (Bound) achieves the lowest NLL on most datasets (e.g., Air 13.58 vs NLL 56.98 and TIC 16.46; Appliances 22.72 vs TIC 30.41) while matching Faithful-level MSE.
• Compute time of 2-W (Bound) stays around 4-6 ms per run across datasets, whereas NLL: TIC takes roughly 45-68 ms; at 32 dimensions TIC needs ~1961 ms and ~22 GB-scale memory versus 2-W bound at ~340 MB.
• Hybrid training on human pose (2-W bound for 20 epochs then NLL with TIC) is reported to retain both low MSE and low NLL, outperforming each component alone.

Measured effects reported in the paper

Surprising findings

• Self-supervised pseudo-labels from a low-dimensional representation did not match the NLL:TIC baseline on human pose, motivating a hybrid schedule.
• Warm-up (training the mean first) did not stabilize NLL and KL-Divergence training; it diverged momentarily on UCI datasets.

Limitations

• Pseudo-labels from kNN over inputs are heuristic and may be inaccurate for high-dimensional inputs such as images, as the authors note for pose.
• The 2-Wasserstein bound is an upper bound rather than the exact distance, and the vanilla 2-Wasserstein formulation was unstable on some datasets (e.g., superconductivity).

Experimental setup

Datasets: synthetic sinusoidal regression synthetic multivariate heteroscedastic data UCI regression (12 datasets) MPII LSP/LSPET

Benchmarks: UCI Regression synthetic sinusoidal synthetic multivariate MPII/LSP human pose

Metrics: MSE NLL TAC KL divergence 2-Wasserstein distance compute time memory

Baselines: NLL KL-Divergence Beta-NLL NLL: Diagonal Faithful NLL: TIC Empirical-Bayes

Builds on: negative log-likelihood heteroscedastic regression TIC (Taylor induced covariance) beta-NLL Faithful heteroscedastic regression ViTPose

Closest papers by meaning

• TIC-TAC: A Framework For Improved Covariance Estimation In Deep Heteroscedastic Regression (ICML 2024 · similarity 0.92)
• Semi-Supervised Regression with Heteroscedastic Pseudo-Labels (NeurIPS 2025 · similarity 0.86)
• The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensing (NeurIPS 2024 · similarity 0.86)
• Pseudo-Labeling for Unsupervised Domain Adaptation with Kernel GLMs (NeurIPS 2026 · similarity 0.85)
• High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.85)
• ConR: Contrastive Regularizer for Deep Imbalanced Regression (ICLR 2024 · similarity 0.85)