Research map / Label-efficient and robust learning / Recognition and long-tail
Gradient Aligned Regression via Pairwise Losses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GAR, a regression loss in label space combining MAE with two pairwise label-difference losses (variance-of-errors and negative Pearson correlation) aggregated via a KL-DRO scheme, achieving linear complexity and strong correlation metrics on tabular and AgeDB benchmarks.
Problem
Regression methods that model label similarity via pairwise latent-space regularization or contrastive learning are computationally expensive (quadratic in batch size) and lack theoretical grounding.
Approach
Match pairwise prediction differences to pairwise label differences directly in label space; show the MSE form reduces to the variance of errors and the normalized form to 1 minus Pearson correlation, giving linear-time losses, then aggregate sub-losses with a KL-DRO log-sum-exp scheme. Theory links pairwise difference matching to matching the ground-truth gradients.
Key results
- GAR achieves the best Pearson and Spearman correlation on all eight real-world tasks, e.g. IC50 Pearson 0.302 vs 0.202 for the next best (ConR).
- On AgeDB from scratch, GAR reaches MAE 6.286 and R^2 0.846 versus previous SOTA MAE 6.40 and R^2 0.830.
- Running time: GAR's loss time is close to MAE/MSE (e.g. 0.003 s vs 0.002 s on Concrete), while RankSim, RNC and ConR are substantially slower (e.g. RNC 0.275 s loss time).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GAR instead of the strongest competing method | regression models · AgeDB (Scratch) | MAE: 6.40 -> 6.286(0.034) | Helped |
| Use GAR instead of the strongest competing method | RNC-pretrained linear probe · AgeDB (RNC Linear Probe) | R²: 0.850 -> 0.854(0.0) | Helped |
| Add pairwise label-difference losses to conventional regression | GAR · real-world regression tasks | Pearson | Helped |
| Add pairwise label-difference losses to conventional regression | GAR · real-world regression tasks | Spearman | Helped |
| Add pairwise MSE difference loss to pointwise MAE (ablation) | GAR · Concrete Compressive Strength | MAE: 5.242(0.212) -> 4.917(0.092) | Helped |
| Add normalized pairwise difference loss to pointwise MAE (ablation) | GAR · Concrete Compressive Strength | MAE: 5.242(0.212) -> 4.621(0.137) | Helped |
| Add both pairwise losses to pointwise MAE (ablation) | GAR · Concrete Compressive Strength | MAE: 5.242(0.212) -> 4.603(0.075) | Helped |
| Remove normalized pairwise difference loss (ablation) | GAR · Concrete Compressive Strength | MAE: 4.603(0.075) -> 4.917(0.092) | Hurt |
| Remove pairwise MSE difference loss (ablation) | GAR · Concrete Compressive Strength | MAE: 4.603(0.075) -> 4.621(0.137) | No clear effect |
| Remove both pairwise losses (ablation) | GAR · Wine Quality | MAE: 0.494(0.004) -> 0.494(0.004) | No clear effect |
| Remove normalized pairwise difference loss (ablation) | GAR · Parkinson (Total) | RMSE: 4.258(0.206) -> 4.214(0.14) | Helped |
| Remove pointwise MAE loss (ablation) | GAR · Super Conductivity | Pearson: 0.949(0.002) -> 0.951(0.001) | Helped |
Surprising findings
- MSE-based pairwise loss and normalized pairwise loss alone each improve MAE, and on Parkinson (Total) the normalized Pearson-based loss alone gives the best Spearman, outperforming full GAR on that metric.
Limitations
- Scope limited to clean data without noise, outliers, or distributional shift; GAR may be misled if training data does not reflect the ground-truth function.
- Sensitive to the robust hyperparameter alpha and batch size; requires tuning.
- Benchmarks are mostly small tabular datasets plus one image dataset; MAE/RMSE gains are modest and not uniform.
Experimental setup
Datasets: Concrete Compressive Strength Wine Quality Parkinson (Total) Parkinson (Motor) Super Conductivity IC50 AgeDB Sine (synthetic) Squared sine (synthetic)
Benchmarks: Concrete Compressive Strength Wine Quality Parkinson (Total) Parkinson (Motor) Super Conductivity IC50 AgeDB
Metrics: MAE RMSE Pearson Spearman R^2
Baselines: MAE MSE Huber Focal (MAE) Focal (MSE) RankSim RNC ConR
Builds on: RankSim Rank-N-Contrast ConR Huber loss Focal loss
Closest papers by meaning
- ConR: Contrastive Regularizer for Deep Imbalanced Regression (ICLR 2024 · similarity 0.87)
- Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks (NeurIPS 2025 · similarity 0.84)
- ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression (AAAI 2025 · similarity 0.84)
- MaxSup: Overcoming Representation Collapse in Label Smoothing (NeurIPS 2025 · similarity 0.84)
- L2B: Learning to Bootstrap Robust Models for Combating Label Noise (CVPR 2024 · similarity 0.84)
- SLACE: A Monotone and Balance-Sensitive Loss Function for Ordinal Regression (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.