Research map / Learning theory and algorithms / Prediction and decision losses

Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Optimal decision tree learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Multistage Defer Trees, a sequence of sparse defer trees that route most samples through one or few trees and defer the rest to a black-box fallback, trained with alternating top-down reweighting.

Problem

Single interpretable trees fall short of black-box accuracy on many tabular datasets, while hybrid interpretable models must defer large fractions of samples to a black box to match its accuracy.

Approach

A multistage chain of sparse defer trees where each stage defers only a small hard region to the next stage, with distance-based sample reweighting so the fallback and later trees borrow strength from neighboring points; compressible into rule lists or single trees.

Key results

• MDT+XGB with at most 25% deferral is within the 95% confidence interval of the best method on most datasets in the accuracy table, and outperforms FIGS on nearly all datasets.
• On California, MDT+XGB matches XGBoost mean accuracy at 25% deferral, which no competing hybrid method achieves within the same budget.
• At around 40% deferral, MDT+XGB comes within 2.0% of XGBoost accuracy on all 24 datasets; no existing method does without deferring the majority of samples.

Surprising findings

• Black-box test error in the deferred region can be more than an order of magnitude higher than the simple model's error in the non-deferred region, motivating distance-based reweighting.

Limitations

• Theoretical optimality guarantees hold only when each component tree is exactly optimal, while the practical LicketySPLIT variant is approximate.
• Deferred samples still rely on a black-box fallback, which is less interpretable for high deferral rates.
• Evaluated only on tabular datasets with a single fallback model type.

Experimental setup

Datasets: Abalone Adult Aging Bank Bike California Churn Droid Heloc Jasmine Phishing Pol

Benchmarks: Abalone Adult Aging Bank Bike California Churn Droid Heloc Jasmine Phishing Pol

Metrics: test accuracy deferral rate average split decisions per sample

Baselines: Non-Deferral Tree FIGS Random Forest XGBoost HyRS Pre-CORELS Post-CORELS FLMM logistic regression deferral

Builds on: LicketySPLIT FIGS HyRS CORELS

Closest papers by meaning

• Feature Learning for Interpretable, Performant Decision Trees (NeurIPS 2023 · similarity 0.90)
• Near-Optimal Decision Trees in a SPLIT Second (ICML 2025 · similarity 0.89)
• Symbolic Regression Enhanced Decision Trees for Classification Tasks (AAAI 2024 · similarity 0.88)
• Leveraging Predictive Equivalence in Decision Trees (ICML 2025 · similarity 0.87)
• Branches: Efficiently Seeking Optimal Sparse Decision Trees via AO* (ICML 2025 · similarity 0.87)
• SORTeD Rashomon Sets of Sparse Decision Trees: Anytime Enumeration (NeurIPS 2025 · similarity 0.87)