Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Regression Trees Know Calculus
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the split structure of a constant-leaf regression tree yields a finite-difference-style gradient estimator (Tree-Based Gradient Estimation) that enables integrated gradients and active subspace estimation for trees.
Problem
Regression trees with constant leaves are piecewise constant and have zero gradient almost everywhere, so gradient-based interpretability and sensitivity tools (integrated gradients, active subspaces) cannot be applied to them directly.
Approach
Compare mean responses of sibling children at each split, scaled by node width along the split variable, to form a finite-difference-like partial derivative estimate; aggregate these down the tree into a gradient estimate (TBGE) computable in one tree traversal, then use it for Monte Carlo and partition-based integro-differential estimators (TBIG, TBAS).
Key results
- On 8 benchmark datasets, TBAS rotation gives the lowest or statistically tied RMSE, e.g. kin40k 0.856 vs 0.963 for identity on depth-4 regression trees, and keggu 0.194 vs 0.35.
- In low-dimensional active subspace estimation, the tree-based estimator forms most of the Pareto front of error versus execution time against GP, PRA, and DASM.
- In high dimension (10, 50, 100) with a sparse 3-coordinate active subspace, TBAS achieves much better accuracy than DASM at significantly lower cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a TBAS rotation instead of PCA | Regression Tree Depth 4 · keggu | 100-fold RMSE: 0.349 -> 0.194 | Helped |
| Use a TBAS rotation instead of PCA | Regression Tree Depth 4 · kin40k | 100-fold RMSE: 0.964 -> 0.856 | Helped |
| Use a TBAS rotation instead of PCA | Regression Tree Depth 4 · concrete | 100-fold RMSE: 0.543 -> 0.47 | Helped |
| Use a TBAS rotation instead of PCA | Regression Tree Depth 4 · grid | 100-fold RMSE: 0.773 -> 0.688 | Helped |
| Use a TBAS rotation instead of PCA | Regression Tree Depth 4 · obesity | 100-fold RMSE: 0.226 -> 0.128 | Helped |
| Use a TBAS rotation instead of no rotation | Regression Tree Depth 4 · bike | 100-fold RMSE: 0.655 -> 0.635 | Helped |
| Use a TBAS rotation instead of no rotation | Regression Tree Depth 4 · supercond | 100-fold RMSE: 0.514 -> 0.511 | No clear effect |
| Use a TBAS rotation instead of PCA | Regression Tree Depth 8 · bike | 100-fold RMSE: 0.407 -> 0.402 | Helped |
| Use a TBAS rotation instead of a random orthogonal rotation | Regression Tree Depth 8 · concrete | 100-fold RMSE: 0.391 -> 0.35 | Helped |
| Use a TBAS rotation instead of no rotation | Regression Tree Depth 8 · keggu | 100-fold RMSE: 0.121 -> 0.078 | Helped |
| Use a TBAS rotation instead of a random orthogonal rotation | Regression Tree Depth 8 · kin40k | 100-fold RMSE: 0.836 -> 0.586 | Helped |
| Use a TBAS rotation instead of PCA | Regression Tree Depth 8 · supercond | 100-fold RMSE: 0.392 -> 0.392 | No clear effect |
Surprising findings
- Gradient estimates from constant-leaf trees are useful even though the tree's formal gradient is zero almost everywhere.
- Theoretical convergence rate of the gradient estimator is slow, about O_P(P (log N)^-1) under the simplified splitting scheme.
Limitations
- Theoretical analysis assumes splits are made on features only (not outcomes), unlike CART, and uses a median-split scheme.
- Convergence is slow, so very deep trees on large datasets are needed for accurate gradients, especially in high dimension.
- Comparison to DASM depends on a particular neural network architecture and optimizer configuration.
Experimental setup
Datasets: bike concrete gas grid keggu kin40k obesity supercond NHEFS MNIST
Benchmarks: bike concrete gas grid keggu kin40k obesity supercond NHEFS MNIST
Metrics: 100-fold RMSE active subspace estimation error (angle) execution time
Baselines: Polynomial Ridge Approximation (PRA) Gaussian process active subspace Deep Active Subspace Method (DASM) PCA rotation random rotation
Builds on: Active Subspace Method Integrated Gradients CART regression trees
Closest papers by meaning
- Learning concise representations for regression by evolving networks of trees (ICLR 2019 · similarity 0.87)
- Smooth And Consistent Probabilistic Regression Trees (NeurIPS 2020 · similarity 0.87)
- A faster training algorithm for regression trees with linear leaves, and an analysis of its complexity (NeurIPS 2025 · similarity 0.87)
- Piecewise Constant and Linear Regression Trees: An Optimal Dynamic Programming Approach (ICML 2024 · similarity 0.86)
- BARK: A Fully Bayesian Tree Kernel for Black-box Optimization (ICML 2025 · similarity 0.84)
- Minimax Rates and Spectral Distillation for Tree Ensembles (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.