Research map / Learning theory and algorithms / Prediction and decision losses
Exploiting Human-AI Dependence for Learning to Defer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a dependent Bayes optimality for learning to defer and a consistent Dependent Cross-Entropy (DCE) surrogate loss that uses the model-expert dependence pattern in training data to decide deferral without explicit confidence estimation.
Problem
Learning to defer (L2D) requires consistent surrogate losses, but the existing Bayes optimality for L2D ignores the dependence between the model's and the expert's correctness, forcing methods to estimate confidences separately.
Approach
Reformulates the L2D Bayes optimality as a dependent form that compares P(Y≠r, Y=M|x) with P(Y=r, Y≠M|x), yielding a deferral principle based on the joint human-model dependence pattern and a DCE loss that establishes the ordering between model and deferral scores directly, while also inducing a bounded expert confidence estimator.
Key results
- On CIFAR-100 with p=94% and k=60, DCE reaches 15.81% error versus 19.33% (OvA) and 18.40% (A-SM)'s budgeted comparisons; DCE achieves lowest misclassification error across all k in both synthetic settings.
- On CIFAR-10N with 'Worse Label' experts, DCE error is 10.69% versus 17.83% for CE; on 'Aggregate Label', 6.34% versus 8.20% for CE.
- On CIFAR-100N, DCE achieves 21.34% error versus 25.61% (CE), 28.50% (A-SM), and 32.07% (OvA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCE instead of the strongest baseline | Learning to Defer on CIFAR-100 with a synthetic expert (p = 94%, k = 60) · CIFAR-100, synthetic expert (p = 94%, k = 60) | misclassification error: 16.22(0.19) -> 15.81(0.31) | Helped |
| Use DCE instead of the strongest baseline | Learning to Defer on CIFAR-100 with a synthetic expert (p = 75%, k = 60) · CIFAR-100, synthetic expert (p = 75%, k = 60) | misclassification error: 21.06(0.54) -> 19.81(0.85) | Helped |
| Use DCE instead of the strongest baseline | Learning to Defer on CIFAR-100N · CIFAR-100N | misclassification error: 25.61(0.52) -> 21.34(0.34) | Helped |
| Use DCE instead of the strongest baseline | Learning to Defer on CIFAR-10N, Worse Label · CIFAR-10N, Worse Label | misclassification error: 17.83(0.28) -> 10.69(0.43) | Helped |
| Use DCE instead of the strongest baseline | Learning to Defer on CIFAR-10N, Aggregate Label · CIFAR-10N, Aggregate Label | misclassification error: 8.20(1.43) -> 6.34(0.21) | Helped |
Limitations
- Experiments use synthetic experts on CIFAR-100 with a simple expert accuracy model and real-world experts on small CIFAR/ImageNet-16H benchmarks
- Consistency is proven for the proposed loss but the paper provides limited analysis of efficiency beyond accuracy and calibration
Experimental setup
Datasets: CIFAR-100 CIFAR-10N CIFAR-100N ImageNet-16H
Benchmarks: CIFAR-100 synthetic experts (p=94%, 75%) CIFAR-100N CIFAR-10N (Worse and Aggregate labels) ImageNet-16H (in appendix)
Metrics: misclassification error budgeted error at 10/20/30% coverage expected calibration error
Baselines: CE (Mozannar & Sontag 2020) OvA (Verma & Nalisnick 2022) A-SM (Cao et al. 2023)
Builds on: Mozannar & Sontag 2020 cross-entropy L2D loss one-versus-all L2D loss (Verma & Nalisnick 2022) asymmetric softmax (Cao et al. 2023) Chow's rule
Closest papers by meaning
- A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems (NeurIPS 2024 · similarity 0.88)
- Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer (NeurIPS 2024 · similarity 0.87)
- A Two-Stage Learning-to-Defer Approach for Multi-Task Learning (ICML 2025 · similarity 0.87)
- Density-Ratio Losses for Post-Hoc Learning to Defer (NeurIPS 2026 · similarity 0.86)
- Regression with Multi-Expert Deferral (ICML 2024 · similarity 0.86)
- Principled Bayesian Optimization in Collaboration with Human Experts (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.