Research map / Optimization / Optimizers
GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRANDE, an end-to-end gradient-descent method learning hard, axis-aligned decision tree ensembles with softsign split activation and instance-wise leaf weighting, evaluated on 19 OpenML binary classification datasets.
Problem
Tabular data remains dominated by tree ensembles (XGBoost, CatBoost); gradient-based methods for tabular data are less accurate and typically use soft or oblique splits that are biased toward smooth functions.
Approach
Extends GradTree's dense straight-through hard-tree formulation from single trees to weighted ensembles, replaces sigmoid/entmoid split functions with a scaled softsign, and learns a per-leaf weight for each estimator so each sample gets instance-wise estimator weights.
Key results
- With optimized hyperparameters GRANDE achieves the highest normalized mean (0.776) and MRR (0.702) macro F1 across 19 datasets, ahead of CatBoost (0.671 / 0.570), XGBoost and NODE.
- With default hyperparameters GRANDE reaches normalized mean 0.6371 and MRR 0.6404, the best among compared methods.
- Ablation shows softsign split activation (normalized mean 0.7906) beats entmoid (0.4188) and sigmoid (0.2207), and leaf-wise weights (0.8235) beat per-estimator weights (0.1765).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GRANDE instead of CatBoost | 19-dataset tabular benchmark with optimized hyperparameters · 19 binary classification datasets | Normalized Mean macro F1: 0.671 -> 0.776 | Helped |
| Use softsign rather than entmoid or sigmoid split activation (ablation) | GRANDE · 19 binary classification datasets | Normalized Mean macro F1: 0.2207 -> 0.7906 | Helped |
| Use instance-wise leaf weights rather than one weight per estimator (ablation) | GRANDE · 19 binary classification datasets | Normalized Mean macro F1: 0.1765 -> 0.8235 | Helped |
Surprising findings
- Per-estimator weighting performs much worse than leaf-based weighting, with normalized mean 0.1765 versus 0.8235.
- With default hyperparameters GRANDE is not best on every dataset; XGBoost and CatBoost win on climate-simulation-crashes and wdbc-style cases.
Limitations
- Shallow ensemble architecture; stacking tree layers and categorical embeddings are left to future work.
- Evaluation restricted to 19 binary classification datasets; multi-class tasks were excluded.
- Slower than GBDT frameworks on GPU, which is highly efficient for XGBoost.
Experimental setup
Datasets: dresses-sales climate-simulation-crashes cylinder-bands wdbc ilpd tokyo1 qsar-biodeg ozone-level-8hr madelon Bioresponse wilt churn
Benchmarks: 19 OpenML binary classification datasets (OpenML benchmark suite selection)
Metrics: macro F1 normalized mean mean reciprocal rank ROC-AUC balanced accuracy runtime
Baselines: XGBoost CatBoost NODE SAINT Random Forest ExtraTrees CART
Builds on: GradTree NODE entmax Anchors
Closest papers by meaning
- Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (ICLR 2020 · similarity 0.89)
- Why do tree-based models still outperform deep learning on typical tabular data? (NeurIPS 2022 · similarity 0.89)
- Neural Tangent Kernels for Axis-Aligned Tree Ensembles (ICML 2024 · similarity 0.86)
- Generative Forests (NeurIPS 2024 · similarity 0.86)
- TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.86)
- Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.