Research map / Learning theory and algorithms / Prediction and decision losses

Generalized additive models via direct optimization of regularized decision stump forests

ICML 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Tree ensemble optimization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Learns fixed-size axis-aligned decision stump forests (GAMs) by direct alternating optimization with ℓ1-roughness and leaf-deviation regularization, without bagging or boosting, matching state-of-the-art GAM accuracy with fewer parameters.

Problem

Learning accurate and interpretable generalized additive models; existing tree-ensemble GAMs rely on boosting or bagging, which do not optimize an explicit objective over a fixed-size model and overfit when stumps are optimized directly.

Approach

Optimize a fixed number of decision stumps directly by alternating exact enumeration per stump with a convex solve for all leaf values, and add a shape-function roughness penalty (ℓ1 discontinuity) plus leaf-deviation-from-bias penalty to control overfitting.

Key results

• On regression benchmarks, ORSF achieves test RMSE competitive with state-of-the-art GAMs (e.g., cpuact 2.37 vs EBM 2.50 and GB 2.43) using far fewer parameters (642 vs 3.4k for GB and 16.6k for EBM).
• On classification, ORSF has the best test error on several datasets, e.g., FICO 27.33% and Churn 19.28%, with smaller models than GB and EBM in most cases.
• Regularized stump forests generalize better than unregularized optimized forests and FastSparse on the cpuact regularization paths.

Measured effects reported in the paper

Surprising findings

• Cubic splines, a traditional statistical baseline, outperform gradient boosting and EBM on many classification datasets.
• Unregularized optimized stump forests overfit badly despite optimizing the training objective well, with best test error at around 200 stumps.

Limitations

• Joint optimization of multiple stump splits is computationally infeasible, so only single stumps are optimized exactly at each step.
• Regularization hyperparameter λ is important and requires tuning on a holdout set.
• Requires a convex solver (MOSEK via CVXPY) for leaf optimization, adding dependency.

Experimental setup

Datasets: cpuact Wine Housing Diamond Year FPS Letter Churn FICO IJCNN Covtype Bank

Benchmarks: cpuact Wine Housing Diamond Year FPS Letter Churn FICO IJCNN Covtype Bank

Metrics: RMSE 0/1 misclassification error model size (number of parameters) training time

Baselines: Explainable Boosting Machine gradient boosting with stumps cubic splines neural additive models FLAM FastSparse

Builds on: Forest Alternating Optimization Tree Alternating Optimization gradient boosting FastSparse Explainable Boosting Machine

Closest papers by meaning

• Literati: Towards Anytime Optimal Shape Generalized Trees via AO* (NeurIPS 2026 · similarity 0.83)
• Empowering Decision Trees via Shape Function Branching (NeurIPS 2025 · similarity 0.83)
• Minimax Rates and Spectral Distillation for Tree Ensembles (NeurIPS 2026 · similarity 0.83)
• Near-Optimal Decision Trees in a SPLIT Second (ICML 2025 · similarity 0.83)
• Branches: Efficiently Seeking Optimal Sparse Decision Trees via AO* (ICML 2025 · similarity 0.83)
• Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again (NeurIPS 2026 · similarity 0.83)