Research map / Learning theory and algorithms / Sample complexity
TIC-TAC: A Framework For Improved Covariance Estimation In Deep Heteroscedastic Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Taylor Induced Covariance (TIC), a closed-form covariance parameterization built from the Jacobian and Hessian of the mean predictor, and Task Agnostic Correlations (TAC), a conditioning-based metric for evaluating predicted covariance, showing improved NLL convergence on synthetic, UCI, and human pose tasks.
Problem
Deep heteroscedastic regression jointly learns mean and covariance via negative log-likelihood, which converges sub-optimally; the predicted covariance is arbitrary without supervision and there is no direct metric to evaluate covariance accuracy without ground truth.
Approach
Model the predictive covariance as a second-order Taylor approximation of the mean under input-neighborhood noise: k1·J·Jᵀ + Hessian-trace term + a learned independent term, all trained through NLL; and evaluate covariance with TAC, the leave-one-out conditional-update absolute error.
Key results
- On UCI regression, NLL-TIC achieves the lowest TAC error on 10 of 12 datasets (e.g., Abalone 1.83 vs 2.85 for best baseline beta-NLL).
- On ViTPose human pose, NLL-TIC reaches average TAC 5.23 and LL -80.31, versus NLL 5.80 / -91.61 and Faithful 5.93 / -91.77.
- In the multivariate synthetic experiment, NLL-TIC's log-likelihood gap to baselines widens with dimensionality; at dim 20 NLL-TIC scores -39.7 vs NLL -67.2 and Faithful -69.0 (higher is better here).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TIC covariance parameterization instead of naive NLL covariance | Deep heteroscedastic regression · Multivariate synthetic, dimensions 4–20 | log-likelihood: -67.2 -> -39.7 | Helped |
| Use TIC covariance parameterization instead of baseline methods | Deep heteroscedastic regression · UCI Regression (12 datasets) | TAC error | Helped |
| Use TIC covariance parameterization instead of baseline methods | ViTPose · MPII + LSP/LSPET human pose estimation | Avg: TAC: 5.80 -> 5.23 | Helped |
| Use TIC covariance parameterization instead of baseline methods | Stacked Hourglass · MPII + LSP/LSPET human pose estimation | Avg: TAC: 6.31 -> 5.64 | Helped |
| Use TIC covariance parameterization | Deep heteroscedastic regression · UCI Regression: Naval | TAC error: 0.20 -> 0.28 | Hurt |
Surprising findings
- On the Naval dataset, the baseline Faithful and NLL outperform TIC, which the authors attribute to low variance in targets giving small gradients.
- Standard NLL-Diagonal and beta-NLL perform very poorly on multivariate pose, with average TAC near 12-13 versus 5-8 for other methods.
Limitations
- Computing the input Hessian is expensive, O(nm^3), limiting scalability.
- Performs poorly when targets have very low variance (Naval dataset), where small gradients hinder the Taylor-based covariance.
- Approximation has no theoretical guarantees and relies on a heuristic neighborhood interpretation.
Experimental setup
Datasets: MPII LSP LSPET UCI regression repository (12 datasets) synthetic sinusoidal synthetic multivariate
Benchmarks: UCI regression (12 datasets) MPII LSP/LSPET synthetic sinusoidal synthetic multivariate
Metrics: TAC error log-likelihood MSE PCKh (referenced)
Baselines: NLL NLL-Diagonal beta-NLL Faithful heteroscedastic regression Natural Laplace MSE
Builds on: negative log-likelihood heteroscedastic regression beta-NLL Faithful heteroscedastic regression Natural Laplace
Closest papers by meaning
- Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression (ICLR 2025 · similarity 0.92)
- The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensing (NeurIPS 2024 · similarity 0.85)
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.85)
- Covariate-Adjusted Deep Causal Learning for Heterogeneous Panel Data Models (NeurIPS 2026 · similarity 0.85)
- Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization (ICML 2025 · similarity 0.84)
- Optimal Nuisance Function Tuning for Estimating a Doubly Robust Functional under Proportional Asymptotics (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.