Research map / Architectures and efficiency / Pruning and efficient tuning

Compressing tree ensembles through Level-wise Optimization and Pruning

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Joint pruning and quantization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LOP, a level-wise method that compresses a trained tree ensemble by pruning subtrees and rescaling/shifting leaf values via L1-regularized logistic regression, tuned to stay within a validation accuracy margin.

Problem

Compressing previously learned tree ensembles (XGBoost, Random Forest) to reduce leaf count, which matters for verification cost, robustness checking, and energy/memory on embedded devices, while keeping predictive accuracy within a margin.

Approach

Process the ensemble level by level from the root; at each level, learn per-node scaling c_n and shift b_n for all leaves below each active node with L1 regularization, so c_n=0 prunes the subtree into a leaf with value b_n, and otherwise leaf values are refit globally. The regularization strength is tuned by binary search in log space on a validation set to meet a maximum balanced-accuracy drop.

Key results

• LOP's average compression ratio is 50.0x on XGBoost models (vs. 33.8x for FP, 7.9x for GR) with average balanced accuracy drop of 0.6 percentage points.
• On RandomForest models LOP averages 1167.2x compression vs. 698.3x for FP, with average balanced accuracy drop of 0.6 points.
• Median robustness-checking time on LOP-compressed XGBoost models is 2.5 to 10 times lower than on competitor-compressed models, and memory footprint is substantially reduced (median 8.2 KB vs. 92.3 KB for original XGBoost).

Measured effects reported in the paper

Surprising findings

• Pareto-optimal compressed models can be obtained from Pareto-suboptimal original XGBoost models, so preselecting models to compress based on their Pareto front does not work.
• A third round of LOP does not change predictive performance much while allowing slightly more compression.

Limitations

• Accuracy-loss margin is enforced on a held-out validation set, so overfitting of the level-wise fits to validation can still occur at deeper levels.
• Evaluation covers only binary classification in the main experiments; regression is in an appendix.
• Compression of tree ensembles is evaluated with balanced accuracy on OpenML datasets only.

Experimental setup

Datasets: Compas Vehicle Spambase Phoneme Adult Ijcnn1 Mnist (2 vs. 4) DryBean (6 vs. rest) Volkert (2 vs. 7) Credit California MiniBooNE

Benchmarks: Compas Vehicle Spambase Phoneme Adult Ijcnn1 Mnist (2 vs. 4) DryBean (6 vs. rest) Volkert (2 vs. 7) Credit California MiniBooNE

Metrics: compression ratio (leaf count) balanced accuracy difference runtime verification time (nearest adversarial example) median splits evaluated memory footprint

Baselines: XGBoost Random Forest Individual Contribution (IC) Global Refinement (GR) LRL1 ForestPrune (FP)

Builds on: Global Refinement Leaf refinement with L1 ensemble pruning ForestPrune Individual Contribution ensemble pruning

Closest papers by meaning

• PINE: Pruning Boosted Tree Ensembles with Conformal In-Distribution Prediction Equivalence (ICML 2026 · similarity 0.90)
• Minimax Rates and Spectral Distillation for Tree Ensembles (NeurIPS 2026 · similarity 0.88)
• Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles (AAAI 2025 · similarity 0.88)
• Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithm (NeurIPS 2025 · similarity 0.85)
• MEPSI: An MDL-Based Ensemble Pruning Approach with Structural Information (AAAI 2024 · similarity 0.85)
• Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again (NeurIPS 2026 · similarity 0.85)