Research map / Optimization / Optimizers
EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EnsLoss, which randomly generates calibrated (convex, classification-calibrated, bounded-below) loss-derivatives at each SGD step so that one training run acts as an ensemble of surrogate losses, reducing overfitting.
Problem
Choosing a single fixed convex calibrated surrogate loss for ERM is hard, and training separate models per loss to ensemble is computationally costly for deep networks; deep models also overfit.
Approach
Transform convexity and classification-calibration conditions into conditions on loss-derivatives, then at each minibatch step sample random RC loss-derivatives (sorted, negative, superlinear raising-tail via inverse Box-Cox sampling) so one training process yields a loss ensemble, analogous to Dropout (doubly stochastic gradients).
Key results
- On CIFAR2 (cat-dog) with ResNet50, EnsLoss reaches 72.04% accuracy vs 67.59% for BCE, 61.57% for Exp, and 67.57% for Hinge.
- On PCam with VGG19, EnsLoss achieves 85.91% accuracy versus 81.58% for BCE and 82.77% for Hinge.
- On 14 OpenML tabular datasets with MLP(5), EnsLoss is significantly better than BCE on 11 datasets and worse on none.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace a fixed surrogate loss with stochastic calibrated loss-derivatives | ResNet models · CIFAR2 (cat-dog) | accuracy: 72.04(0.35) vs 67.57(0.28) | Helped |
| replace a fixed surrogate loss with stochastic calibrated loss-derivatives | VGG models · PCam | accuracy: 85.91(0.19) vs 82.77(0.41) | Helped |
| replace fixed losses with EnsLoss | VGG16 · 45 CIFAR2 datasets | accuracy | Helped |
| randomly update inverse Box-Cox λ over epochs instead of keeping λ=0 fixed (ablation) | EnsLoss with ResNet50 · CIFAR2 datasets | accuracy: 70.04(1.21) -> 70.87(0.72); 81.12(0.28) -> 80.45(0.30); 83.11(0.29) -> 83.05(0.12) | Mixed |
Surprising findings
- Random loss-derivatives from a fixed inverse Box-Cox transform (lambda=0) work better than tuning lambda; randomly updating lambda gave no significant gains.
- EnsLoss often requires more epochs than fixed-loss methods to reach maximum training accuracy, with longer training time.
Limitations
- Requires more epochs and longer training time than fixed losses.
- Choice of the positive random variable (inverse Box-Cox parameter) needs further investigation.
- Distribution-free excess risk bound for the ensemble loss appears suboptimal.
Experimental setup
Datasets: OpenML tabular datasets (14) CIFAR10 (45 pairwise CIFAR2 subsets) PatchCamelyon (PCam)
Benchmarks: OpenML tabular datasets (14) CIFAR2 (45 pairwise subsets) PCam
Metrics: classification accuracy AUC
Baselines: BCE (logistic loss) Hinge loss Exponential loss Dropout weight decay data augmentation
Builds on: Dropout SGD classification-calibration theory (Bartlett et al.) Box-Cox transformation
Closest papers by meaning
- Weighted Ensemble Self-Supervised Learning (ICLR 2023 · similarity 0.86)
- Generalized Ambiguity Decompositions for Classification with Applications in Active Learning and Unsupervised Ensemble Pruning (AAAI 2017 · similarity 0.86)
- Subsampled Ensemble Can Improve Generalization Tail Exponentially (NeurIPS 2025 · similarity 0.84)
- Structure-informed Risk Minimization for Robust Ensemble Learning (ICML 2025 · similarity 0.84)
- Trading off Consistency and Dimensionality of Convex Surrogates for Multiclass Classification (NeurIPS 2024 · similarity 0.84)
- Loss Functions and Operators Generated by f-Divergences (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.