Research map / Learning theory and algorithms / Prediction and decision losses
Consistent Estimators for Learning to Defer to an Expert
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use L_CE imputation of expert agreement labels for unlabeled-expert examples rather than the two-step training strategy (ablation) | CIFAR-10H · CIFAR-10H | system accuracy: 96.03±0.21 -> 96.29±0.25 | Helped |
| use L_CE imputation of expert agreement labels rather than Confidence | CIFAR-10H · CIFAR-10H | system accuracy: 95.09±0.40 -> 96.29±0.25 | Helped |
| train L_CE with alpha 0 instead of alpha 1 or 0.5 (ablation) | L_CE on the synthetic Gaussian mixture · Synthetic Gaussian mixture | system accuracy: +6.39 vs Confidence; +0.22 vs Oracle; +2.01 vs MixOfExp | Mixed |
| reduce alpha from 1 to 0.5 (ablation) | L_CE on CIFAR-10 · CIFAR-10 | system accuracy: 90.41 -> 90.92 (expert k=1); 97.70 -> 99 (expert k=9) | Helped |
| use joint L_CE^1 training instead of the Confidence baseline | CIFAR-10, expert k=5 · CIFAR-10, 2,000 training points | system accuracy: 62.33% -> 70.12% | Helped |
| use L_CE^1 rather than Confidence | CIFAR-100 · CIFAR-100, 30 <= k <= 90 | test accuracy: 1.60 | Helped |
| use L_CE^1 rather than Confidence | Hate Speech and Offensive Language Detection · AAE-biased expert | system accuracy: 92.42±0.40 -> 92.91±0.17 | Helped |
| use L_CE^1 rather than Confidence | Hate Speech and Offensive Language Detection · non-AAE-biased expert | system accuracy: 90.60 ±0.13 -> 90.42 ± 0.38 | Hurt |
| jointly model classifier and expert deferral rather than using confidence scores | CheXpert · CheXpert, expert q=0.7 and p=1 | maximum AU-ROC: 0.043, 0.029, 0.016, 0.022 and 0.025 | Helped |
| use L_CE^1 rather than Confidence as training data grows | CIFAR-10 classifier · CIFAR-10, expert k=5 | classifier accuracy on non-deferred examples: 62.84 vs 50.31; 71.51 vs 59; 95.42 vs 92.45 | Mixed |
| use confidence-score deferral with a fair expert rather than the classifier alone | Hate Speech and Offensive Language Detection · Fair expert | system discrimination: 0.226 -> 0.449 | Hurt |
| train the mixture-of-experts baseline through the end of training rather than use the consistent L_CE approach | CIFAR-10 · CIFAR-10, expert k<8 | deferral behavior | Hurt |
Closest papers by meaning
- Exploiting Human-AI Dependence for Learning to Defer (ICML 2024 · similarity 0.88)
- Regression with Multi-Expert Deferral (ICML 2024 · similarity 0.86)
- A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems (NeurIPS 2024 · similarity 0.85)
- When Lower-Order Terms Dominate: Adaptive Expert Algorithms for Heavy-Tailed Losses (NeurIPS 2025 · similarity 0.85)
- When to Act and When to Ask: Policy Learning With Deferral Under Hidden Confounding (NeurIPS 2024 · similarity 0.85)
- Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.