Research map / Learning theory and algorithms / Prediction and decision losses
Mastering Multiple-Expert Routing: Realizable $H$-Consistency and Strong Guarantees for Learning to Defer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes new comp-sum-based surrogate losses for single-stage and two-stage learning to defer with multiple experts, proving realizable H-consistency, H-consistency bounds, and Bayes-consistency, plus low-noise refinements, with experiments on CIFAR-10/100, SVHN, and Tiny ImageNet.
Problem
Learning to defer with multiple experts requires surrogate losses that are consistent with the deferral loss; prior multiple-expert surrogates lack realizable H-consistency, and two-stage surrogates lack it for classification-error-based costs.
Approach
Reformulates the single-stage deferral loss via indicator identities, then replaces indicators with comp-sum surrogates (including Psi_q and MAE variants) to get losses that are realizable H-consistent and admit H-consistency bounds; for two-stage, proposes a cost-sensitive margin-based surrogate (logistic/comp-sum style over expert scores) with analogous guarantees, and adds Tsybakov/Massart low-
Key results
- Single-stage, CIFAR-10: the proposed loss reaches 92.21% system accuracy vs. 91.85% and 91.98% for the baselines, with a deferral ratio of 61.2% (vs. 37.1% and 33.4% for the baselines).
- Two-stage, CIFAR-10: the proposed loss reaches 93.34% system accuracy vs. 92.92% for the baseline.
- On a realizable synthetic Mixture-of-Gaussians dataset, the proposed losses (q=0.7 and q=1) reach close to 100% system accuracy while baselines fail to find near-zero-error solutions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed single-stage surrogate with Ψ(u) = 1 - u instead of the strongest baseline | Single-stage multiple-expert deferral · CIFAR-10 | system accuracy: 91.98 $\pm$ 0.09 -> 92.21 $\pm 0.18$ | No clear effect |
| Use the proposed single-stage surrogate with Ψ(u) = 1 - u instead of the strongest baseline | Single-stage multiple-expert deferral · CIFAR-100 | system accuracy: 64.93 $\pm 0.22 -> 65.39 $\pm 0.31$ | No clear effect |
| Use the proposed single-stage surrogate with Ψ(u) = 1 - u instead of the strongest baseline | Single-stage multiple-expert deferral · SVHN | system accuracy: 95.91 $\pm 0.08 -> 96.08 $\pm 0.11$ | No clear effect |
| Use the proposed single-stage surrogate with Ψ(u) = 1 - u instead of the strongest baseline | Single-stage multiple-expert deferral · Tiny ImageNet | system accuracy: 50.89 $\pm 0.49 -> 51.13 $\pm 0.56$ | No clear effect |
| Use the proposed two-stage surrogate with Ψ(u) = -log(u) instead of the baseline | Two-stage multiple-expert deferral · CIFAR-10 | system accuracy: 92.92 $\pm$ 0.14 -> 93.34 $\pm 0.18$ | No clear effect |
| Use the proposed two-stage surrogate with Ψ(u) = -log(u) instead of the baseline | Two-stage multiple-expert deferral · CIFAR-100 | system accuracy: 67.99 $\pm$ 0.19 -> 68.70 $\pm 0.23$ | No clear effect |
| Use the proposed two-stage surrogate with Ψ(u) = -log(u) instead of the baseline | Two-stage multiple-expert deferral · SVHN | system accuracy: 96.30 $\pm$ 0.06 -> 96.47 $\pm 0.08$ | No clear effect |
| Use the proposed two-stage surrogate with Ψ(u) = -log(u) instead of the baseline | Two-stage multiple-expert deferral · Tiny ImageNet | system accuracy: 45.81 $\pm$ 0.17 -> 46.62 $\pm 0.24$ | No clear effect |
| Use the proposed surrogate with Ψq for q = 0.7 or q = 1 rather than the compared baseline losses | Single-stage multiple-expert deferral on realizable Mixture-of-Gaussians · Mixture-of-Gaussians | system accuracy: close to 100% | Helped |
Surprising findings
- In the single-stage experiments the proposed loss defers substantially more than baselines (e.g., 61.2% vs. about 33-37% on CIFAR-10) while achieving comparable accuracy.
Limitations
- Experiments are limited to image benchmarks with synthetic expert domains; real-world overlapping experts and heterogeneous data are left to future work.
- Assumes expert predictions are available during training.
- Non-realizable experiments only show parity with baselines rather than improvement.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN Tiny ImageNet Mixture-of-Gaussians (synthetic)
Benchmarks: CIFAR-10 CIFAR-100 SVHN Tiny ImageNet
Metrics: system accuracy deferral ratio
Baselines: single-stage multiple-expert surrogate losses (prior work) two-stage multiple-expert surrogate losses (prior work)
Builds on: comp-sum losses realizable H-consistency (RL2D) H-consistency bounds
Closest papers by meaning
- Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer (NeurIPS 2024 · similarity 0.90)
- Regression with Multi-Expert Deferral (ICML 2024 · similarity 0.88)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.87)
- A Universal Growth Rate for Learning with Smooth Surrogate Losses (NeurIPS 2024 · similarity 0.86)
- Learning to Route Among Specialized Experts for Zero-Shot Generalization (ICML 2024 · similarity 0.85)
- Exploiting Human-AI Dependence for Learning to Defer (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.