Research map / Learning theory and algorithms / Prediction and decision losses
Consistency of surrogate losses
Prove calibration and consistency bounds for convex surrogates of classification losses.
72 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Prediction and decision losses, keyword cluster “losses · loss functions · surrogate”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 11 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.4% to 1.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Asymmetric Multi-task Learning Based on Task Relatedness and Loss (ICML 2016 · 82 citations)
- 2017 · Revenue Optimization with Approximate Bid Predictions (NeurIPS 2017 · 52 citations)
- 2018 · Algorithms and Theory for Multiple-Source Adaptation (NeurIPS 2018 · 122 citations)
- 2019 · A Theoretically Sound Upper Bound on the Triplet Loss for Improving the Efficiency of Deep Distance Metric Learning (CVPR 2019 · 62 citations)
- 2020 · Online Active Learning of Reject Option Classifiers (AAAI 2020 · 11 citations)
- 2021 · On UMAP's True Loss Function (NeurIPS 2021 · 14 citations)
- 2022 · Fisher SAM: Information Geometry and Sharpness Aware Minimisation (ICML 2022 · 6 citations)
- 2023 · Adaptive Annealing for Robust Geometric Estimation (CVPR 2023 · 6 citations)
- 2024 · Multi-Label Learning with Stronger Consistency Guarantees (NeurIPS 2024 · 4 citations)
- 2025 · Loss Functions and Operators Generated by f-Divergences (ICML 2025 · 0 citations)
- 2026 · Networked Information Aggregation for Binary Classification (ICML 2026 · 0 citations)
Earliest papers here
- Asymmetric Multi-task Learning Based on Task Relatedness and Loss (ICML 2016 · 82 citations)
- A Minimax Approach to Supervised Learning (NeurIPS 2016 · 56 citations)
- Optimal Black-Box Reductions Between Optimization Objectives (NeurIPS 2016 · 50 citations)
What has been tried
59 papers here report 492 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 13%, hurt 18%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use Realizable L2D with q = 0.7 instead of q = 1 | HateSpeech · system accuracy | 91.97 ± 0.29 -> 92.20 ± 0.54 | Mixed | Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer (NeurIPS 2024 · 3 citations) |
| Use linear cardinality cost k instead of logarithmic cost log(k) | CIFAR-10, CIFAR-100, ImageNet, and SVHN · accuracy versus cardinality | No clear effect | Cardinality-Aware Set Prediction and Top-$k$ Classification (NeurIPS 2024 · 3 citations) | |
| Replace backward differencing (PGB) with central differencing (PGC) | Simple misspecification experiment · normalized excess regret | Helped | Decision-Focused Learning with Directional Gradients (NeurIPS 2024 · 3 citations) | |
| Increase sampled perturbations from s=3 to s=5 | CIFAR100 · test accuracy | 77.40 -> 77.78 | Helped | Tilted Sharpness-Aware Minimization (ICML 2025 · 0 citations) |
| Use randomized solution caching during SPO-RC+ training | Fractional knapsack, ℓ1 conformity score · NormSPORCTest | Hurt | Smart Surrogate Losses for Contextual Stochastic Linear Optimization with Robust Constraints (NeurIPS 2025 · 0 citations) | |
| refine the original 8-ray fan by adding one ray | Two-dimensional synthetic data · accuracy | Helped | How to Learn a Star: Binary Classification with Starshaped Polyhedral Sets (NeurIPS 2025 · 0 citations) | |
| Use tanh instead of the -log lower bound | CIFAR-10 test, 5-model average · accuracy | 95.88 (± 0.08) -> 95.93 (± 0.08) | Helped | Improving SAM Requires Rethinking its Optimization Formulation (ICML 2024 · 0 citations) |
| Set threshold epsilon to 0.01 or 0.05 instead of the selected 0.1 | Ant Morphology and TF-BIND-10 · performance improvement over baseline | Hurt | Incorporating Surrogate Gradient Norm to Improve Offline Optimization Techniques (NeurIPS 2024 · 0 citations) | |
| Increase network depth from 3 to 5 layers | Yeast · F-measure | 0.5903 -> 0.4951 | Hurt | Stabilizing Sample Similarity in Representation via Mitigating Random Consistency (ICML 2025) |
| vary initial training epochs R (5, 10, 20, 30) | CIFAR-100 and Tiny-ImageNet, 70% and 30% selection · accuracy | Mixed | Efficient Core-set Selection for Deep Learning Through Squared Loss Minimization (ICML 2025) | |
| Replace the elastic-net penalty with MCP | SVM simulation, p = 10000, ρ = 0.2 · test error | 0.11(0.02) -> 0.10(0.01) | Helped | Finite Smoothing Algorithm for High-Dimensional Support Vector Machines and Quantile Regression (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 2 of 10 papers
- Tiny-ImageNet · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 29 papers; from OpenAlex and the papers' affiliations).
- Google · 9 papers
- Courant Institute of Mathematical Sciences · 6 papers
- Stanford University · 3 papers
- University of Chicago · 2 papers
- Google DeepMind · 2 papers
- University of Southern California · 2 papers
- University of Colorado at Boulder · 2 papers
- University of Colorado Boulder · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Losses for decision-making (68 papers)
- Loss landscape analysis (61 papers)
- Ranking-based loss optimization (50 papers)
Elsewhere in the topic
- Adaptive testing and experiments (201 papers)
- Active learning and subset selection (199 papers)
- Distributionally robust optimization (184 papers)
- Data attribution and recourse (160 papers)
- Theory of neural network sensitivity (143 papers)
- Ranking models from comparisons (127 papers)
- Determinantal point process inference (122 papers)
- Conformal prediction sets (116 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).