Research map / Learning theory and algorithms / Combinatorial and search optimization
A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Learn-to-defer systems must optimize human-AI team accuracy while satisfying multiple constraints (fairness, expert budget, OOD deferral, etc.), but general methodology for such multi-objective constrained deferral has been largely unexplored.
Approach
Reformulates multi-objective learn-to-defer as a linear program over randomized predictors and shows it is an instance of a d-dimensional generalization of the Neyman-Pearson lemma (d-GNP), yielding a closed-form optimal solution of thresholded embedding scores that a plug-in post-processing algorithm can estimate.
Key results
- On COMPAS, d-GNP achieves better demographic parity than baselines Madras et al. and Mozannar et al., averaged over 10 random seeds, though not always better accuracy.
- On ACSIncome with a simulated expert, the method's test-set equalized odds violation stays within the specified bounds, with accuracy increasing as bounds are loosened.
- The paper proves d-GNP optimality and a generalization rate of O(sqrt(log n / n)) for constraints in the single-constraint setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the d-GNP post-processing method instead of the compared baselines | Learn-to-Defer on COMPAS · COMPAS | demographic parity violation | Helped |
| loosen the equalized-odds violation bounds | d-GNP on ACSIncome · ACSIncome | accuracy | Helped |
| apply the d-GNP post-processing method under bounded equalized odds violation | Learn-to-Defer on ACSIncome · ACSIncome | equalized odds violation | Helped |
Surprising findings
- Staged learning (train classifier, then fit deferral) is non-compositional for fairness: a fair classifier and a fair expert can yield an unfair deferral system.
- Finding the optimal deterministic classifier and rejector under a bounded expert budget is NP-hard for finite input spaces, which motivates randomized solutions.
Limitations
- Generalization guarantees are proven only for the single-constraint case; multiple-constraint extension left to future work.
- Experiments use only two tabular datasets, and human expert decisions on COMPAS are real but on ACSIncome simulated.
Experimental setup
Datasets: COMPAS ACSIncome (folktables)
Benchmarks: COMPAS ACSIncome
Metrics: accuracy demographic parity violation equalized odds violation constraint violation
Baselines: Madras et al. (demographic parity regularizer) Mozannar et al. (score-shifting post-hoc thresholding)
Builds on: Neyman-Pearson lemma Chow's rule Madras et al. learning to defer Mozannar and Sontag consistent estimators for learning to defer
Closest papers by meaning
- Exploiting Human-AI Dependence for Learning to Defer (ICML 2024 · similarity 0.88)
- Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision (NeurIPS 2024 · similarity 0.85)
- A Two-Stage Learning-to-Defer Approach for Multi-Task Learning (ICML 2025 · similarity 0.85)
- Principled Bayesian Optimization in Collaboration with Human Experts (NeurIPS 2024 · similarity 0.83)
- Density-Ratio Losses for Post-Hoc Learning to Defer (NeurIPS 2026 · similarity 0.83)
- Regression with Multi-Expert Deferral (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.