Research map / Label-efficient and robust learning / Recognition and long-tail

Improving Deep Regression with Tightness

ICLR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that preserving target ordinality reduces conditional entropy H(Z|Y) (representation tightness), argues standard regressors tighten poorly, and proposes multiple-target duplication plus an optimal-transport regularizer (ROT-Reg) to improve deep regression.

Problem

Deep regression representations are often not ordinal or tight with respect to targets, and typical regression losses do little to reduce the conditional entropy H(Z|Y), which hurts generalization.

Approach

Links target ordinality to representation tightness (low H(Z|Y)); gradient analysis shows linear regressors can only tighten representations along the regressor direction, so duplicating regression targets adds constraints (MT), and a self-entropic optimal transport regularizer matches local similarity structure between target and feature spaces (ROT-Reg).

Key results

• On AgeDB-DIR, MT + L_ot reduces overall MAE from 7.80 (baseline) to 7.28 with ten-run standard deviations.
• On NYUD2-DIR, MT + L_ot reaches RMSE 1.321 versus 1.477 for the baseline, and the best delta_1 of 0.715 comes from MT alone.
• On coordinate prediction (Torus), L_ot alone lowers MSE from 3.01 to 0.77 and MT + L_ot reaches 0.75.

Measured effects reported in the paper

Surprising findings

• Standard regression yields low feature-target ordinality (Spearman 0.39 cosine) and RankSim with cosine distance collapses representations to near-zero volume, which the authors note as a contrast with the tightness goal.
• Gradient analysis indicates with a fixed linear regressor that feature updates only follow the regressor direction, so regression tightens representations weakly compared with classification.

Limitations

• Tightness analysis relies on gradient/linear-regressor assumptions and a convexity assumption on solution sets.
• Choosing gamma too low (e.g., 0.01) leads to NaN values in Sinkhorn computation; hyperparameters lambda and gamma require tuning.
• Experiments limited to three tasks with ResNet-50-based baselines; the number of targets M depends on intrinsic dimension and is not automatically determined.

Experimental setup

Datasets: AgeDB-DIR NYUD2-DIR coordinate prediction manifolds (Mammoth, Torus, Circle)

Benchmarks: AgeDB-DIR NYUD2-DIR coordinate prediction

Metrics: MAE RMSE delta_1 accuracy MSE Spearman's rank correlation Kendall's rank correlation

Baselines: RankSim Ordinal Entropy (OE) PH-Reg InfDrop Topological Autoencoder

Builds on: RankSim Ordinal Entropy PH-Reg (topological autoencoder) self-entropic optimal transport Deep imbalanced regression (DIR)

Closest papers by meaning

• Improving Deep Regression with Ordinal Entropy (ICLR 2023 · similarity 0.89)
• Deep Regression Representation Learning with Topology (ICML 2024 · similarity 0.86)
• Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.85)
• Batch Normalization Is Blind to the First and Second Derivatives of the Loss (AAAI 2024 · similarity 0.84)
• High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.83)
• $H$-Consistency Guarantees for Regression (ICML 2024 · similarity 0.83)