Research map / Learning theory and algorithms / Prediction and decision losses
Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a family of comp-sum-based surrogate losses for learning to defer parameterized by a non-increasing Psi, proving realizable H-consistency, H-consistency bounds for symmetric complete hypothesis sets, and Bayes-consistency including resolving an open question about the RS surrogate.
Problem
Surrogate losses for learning to defer (L2D) need guarantees (Bayes-consistency, realizable H-consistency, H-consistency bounds) simultaneously, and prior surrogates satisfied only subsets of these, mostly for classification-error cost functions.
Approach
Derive a family of L2D surrogates by replacing the two indicator terms of the deferral loss with comp-sum losses, where the second term uses the modified softmax numerator e^{h(x,y)}+e^{h(x,n+1)}; choosing Psi appropriately gives realizable H-consistency, H-consistency bounds, and Bayes-consistency, including for general cost functions when Psi(t)=1-t.
Key results
- On HateSpeech, Realizable L2D with q=0.7 reaches 92.20 system accuracy versus 92.18 for the best baseline L_OvA, i.e. comparable.
- On CIFAR-10H, Realizable L2D with q=0.7 reaches 96.80 system accuracy, the best among methods (baseline best 96.75 for L_general).
- On COMPASS, Realizable L2D with q=1 reaches 66.83 system accuracy versus 66.67 for the best baseline L_general, a small margin with overlapping standard deviations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Realizable L2D with q = 0.7 instead of q = 1 (ablation) | Realizable L2D · HateSpeech | system accuracy: 91.97 ± 0.29 -> 92.20 ± 0.54 | Mixed |
| Use Realizable L2D with q = 0.7 instead of q = 1 (ablation) | Realizable L2D · COMPASS | system accuracy: 66.83 ± 0.85 -> 66.17 ± 2.01 | Hurt |
| Use Realizable L2D with q = 0.7 instead of q = 1 (ablation) | Realizable L2D · CIFAR-10H | system accuracy: 96.57 ± 0.05 -> 96.80 ± 0.25 | Helped |
| Use Realizable L2D with q = 0.7 instead of the strongest system-accuracy baseline | Learning to defer · HateSpeech | system accuracy: 92.18 ± 0.10 -> 92.20 ± 0.54 | Helped |
| Use Realizable L2D with q = 1 instead of the strongest system-accuracy baseline | Learning to defer · COMPASS | system accuracy: 66.67 ± 0.62 -> 66.83 ± 0.85 | Helped |
| Use Realizable L2D with q = 0.7 instead of the strongest system-accuracy baseline | Learning to defer · CIFAR-10H | system accuracy: 96.75 ± 0.55 -> 96.80 ± 0.25 | Helped |
Surprising findings
- Realizable H-consistency does not imply H-consistency bounds in L2D, unlike standard classification where bounds imply realizable consistency.
- The loss L_CE has H-consistency bounds but is not realizable H-consistent.
Limitations
- Consistency bounds for classification-error costs require symmetric and complete hypothesis sets; the general-cost bound needs the MAE choice of Psi.
- Analysis restricted to single-stage, single-expert setting.
- Experimental gains over baselines are small and often within standard deviations.
Experimental setup
Datasets: CIFAR-10H HateSpeech COMPASS
Benchmarks: CIFAR-10H HateSpeech COMPASS
Metrics: system accuracy accepted accuracy coverage
Baselines: L_CE (Mozannar and Sontag cross-entropy surrogate) L_OvA (one-vs-all surrogate) L_RS (realizable surrogate) L_general (general cost surrogate)
Builds on: comp-sum losses realizable surrogate loss of Verma and Nalisnick (RS loss) H-consistency bounds generalized cross entropy loss
Closest papers by meaning
- A Universal Growth Rate for Learning with Smooth Surrogate Losses (NeurIPS 2024 · similarity 0.91)
- $H$-Consistency Guarantees for Regression (ICML 2024 · similarity 0.90)
- Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer (ICML 2025 · similarity 0.90)
- Trading off Consistency and Dimensionality of Convex Surrogates for Multiclass Classification (NeurIPS 2024 · similarity 0.90)
- Exploiting Human-AI Dependence for Learning to Defer (ICML 2024 · similarity 0.87)
- Consistency Conditions for Differentiable Surrogate Losses (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.