Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Statistical Inference for Gradient Boosting Regression

NeurIPS 2025 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Regression uncertainty quantification · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends Boulevard-regularized boosting with dropout and parallel leave-one-out training to obtain CLTs, enabling built-in confidence/prediction/reproduction intervals and chi-squared variable importance tests, with Nyström sketching for scalability.

Problem

Statistical inference and uncertainty quantification (confidence intervals, prediction intervals, hypothesis tests) for gradient boosting regression, which lacks principled frequentist guarantees and where the existing Boulevard regularization recovers at most half the signal.

Approach

Combine random dropout (BRAT-D) or parallel tree growth with leave-one-out residuals (BRAT-P) into Boulevard regularization so the ensemble converges to a kernel ridge regression with improved signal recovery and a CLT, which yields built-in intervals and tests; higher dropout recovers more signal.

Key results

• Dropout-based and parallel algorithms achieve asymptotic relative efficiency gains over vanilla Boulevard of up to 4x (BRAT-D) and at least 4x (BRAT-P) per the theory.
• Algorithms are competitive in test MSE with XGBoost, LightGBM, and random forests across nine UCI datasets, interpolating between boosting and random forests.
• Prediction intervals attain near-nominal coverage after adaptive calibration-based adjustment, while reproduction intervals and conditional confidence coverage underperform.

Measured effects reported in the paper

Surprising findings

• Increasing the dropout rate recovers more of the signal, the opposite of the usual intuition that dropout only regularizes.
• Boulevard-style regularization without rescaling is needed for CLTs; the proposed methods avoid the factor-of-two signal loss of vanilla Boulevard.

Limitations

• CLTs rely on structure-value isolation (integrity/honesty) and non-adaptivity assumptions that experiments relax but theory requires.
• Convergence rate is quadratically worse than the Lipschitz minimax rate and matches only the 1/2-Hölder rate.
• Reproduction and conditional CI coverage fall short of nominal levels in experiments.

Experimental setup

Datasets: UCI Machine Learning Repository datasets (nine, including Wine Quality, Air Quality, Abalone) Friedman function (synthetic) simulated function sin(2πx)+x²/2

Benchmarks: UCI nine datasets (test MSE) Friedman function coverage simulated variable importance tests

Metrics: test MSE marginal coverage conditional coverage interval width size power

Baselines: XGBoost LightGBM random forests vanilla gradient boosting vanilla Boulevard elastic net conformal prediction

Builds on: Boulevard regularization random dropout parallel backfitting Nyström approximation reproduction intervals

Closest papers by meaning

• Regularized Gradient Boosting (NeurIPS 2019 · similarity 0.88)
• Gradient Boosting Performs Gaussian Process Inference (ICLR 2023 · similarity 0.88)
• Variational Boosting: Iteratively Refining Posterior Approximations (ICML 2017 · similarity 0.88)
• Revisiting Agnostic Boosting (NeurIPS 2025 · similarity 0.86)
• Gradient Regularized Newton Boosting Trees with Global Convergence (NeurIPS 2026 · similarity 0.86)
• Gradient Descent Converges Arbitrarily Fast for Logistic Regression via Large and Adaptive Stepsizes (ICML 2025 · similarity 0.85)