Research map / Label-efficient and robust learning / Recognition and long-tail
Dist Loss: Enhancing Regression in Few-Shot Region through Distribution Distance Constraint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dist Loss, a loss that adds a differentiable batch-level distribution-distance term (KDE-derived pseudo-labels vs. sorted predictions via fast differentiable sorting) to sample-wise error to reduce few-shot errors in deep imbalanced regression.
Problem
Deep regression models trained on imbalanced label distributions bias predictions toward many-shot regions, yielding large errors in few-shot label regions where rare cases (e.g., abnormal potassium) matter most.
Approach
Align the batch prediction distribution with the KDE-estimated label distribution by comparing sorted predictions with KDE-sampled pseudo-labels using a differentiable sorting operator, and jointly optimize this with sample-wise L1/L2 error; applied as a fine-tuning of the linear layer of a pretrained model.
Key results
- On few-shot region, Dist Loss reaches MAE 22.550 (IMDB-WIKI-DIR), 9.122 (AgeDB-DIR), and 1.329 (ECG-K-DIR), versus 26.930, 12.894, 1.771 for vanilla.
- Dist Loss added to LDS, FDS, Ranksim, ConR, or Balanced MSE improves few-shot MAE in most cases; e.g., LDS+Dist Loss gives 22.331 / 10.437 / 1.325 MAE.
- Dist Loss adds little training time over Balanced MSE (154.0 s vs 152.6 s per epoch on IMDB-WIKI-DIR) while feature-level methods are much slower.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Dist Loss to the vanilla model | ResNet-50 · IMDB-WIKI-DIR few-shot | MAE: 26.930 -> 22.550 | Helped |
| add Dist Loss to the vanilla model | ResNet-50 · AgeDB-DIR few-shot | MAE: 12.894 -> 9.122 | Helped |
| add Dist Loss to the vanilla model | Net1D · ECG-K-DIR few-shot | MAE: 1.771 -> 1.329 | Helped |
| add Dist Loss to the vanilla model | ResNet-50 · IMDB-WIKI-DIR few-shot | GM: 21.254 -> 13.752 | Hurt |
| add Dist Loss to the vanilla model | ResNet-50 · AgeDB-DIR few-shot | GM: 9.789 -> 5.453 | Helped |
| add Dist Loss to the vanilla model | Net1D · ECG-K-DIR few-shot | GM: 1.578 -> 0.978 | Helped |
| add Dist Loss to LDS (ablation) | LDS · IMDB-WIKI-DIR few-shot | MAE: 22.753 -> 22.331 | Helped |
| add Dist Loss to LDS (ablation) | LDS · AgeDB-DIR few-shot | MAE: 11.279 -> 10.437 | Helped |
| add Dist Loss to LDS (ablation) | LDS · ECG-K-DIR few-shot | MAE: 1.510 -> 1.325 | Helped |
| add Dist Loss to Balanced MSE (ablation) | Balanced MSE · IMDB-WIKI-DIR few-shot | GM: 12.603 -> 14.238 | Hurt |
| use batch size 256 instead of 512 (ablation) | Dist Loss · IMDB-WIKI-DIR few-shot | MAE: 22.516 -> 22.323 | No clear effect |
| use batch size 768 instead of 512 (ablation) | Dist Loss · AgeDB-DIR few-shot | MAE: 9.122 -> 9.148 | No clear effect |
Surprising findings
- Dist Loss underperforms Balanced MSE on the few-shot GM metric for IMDB-WIKI-DIR (14.288 vs 12.603), which the authors attribute to GM's multiplicative sensitivity to small frequent errors.
- Batch size (256 to 768) has negligible effect on performance.
Limitations
- On the GM metric in few-shot regions for IMDB-WIKI-DIR, Dist Loss is worse than Balanced MSE.
- Gains are reported on regression with a fixed bin discretization and KDE; generality to other tasks is not tested beyond three datasets.
- Method depends on fine-tuning the linear layer of a pretrained vanilla model rather than full end-to-end training.
Experimental setup
Datasets: IMDB-WIKI-DIR AgeDB-DIR ECG-K-DIR (from MIMIC-IV)
Benchmarks: IMDB-WIKI-DIR AgeDB-DIR ECG-K-DIR
Metrics: MAE GM
Baselines: LDS FDS Ranksim ConR Balanced MSE
Builds on: Balanced MSE Label Distribution Smoothing Feature Distribution Smoothing Fast differentiable sorting and ranking
Closest papers by meaning
- ConR: Contrastive Regularizer for Deep Imbalanced Regression (ICLR 2024 · similarity 0.85)
- A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation (AAAI 2025 · similarity 0.84)
- Leveraging Group Classification with Descending Soft Labeling for Deep Imbalanced Regression (AAAI 2025 · similarity 0.84)
- Improve Representation for Imbalanced Regression through Geometric Constraints (CVPR 2025 · similarity 0.84)
- Deep Imbalanced Regression via Hierarchical Classification Adjustment (CVPR 2024 · similarity 0.84)
- Pooling Image Datasets with Multiple Covariate Shift and Imbalance (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.