Research map / Learning theory and algorithms / Kernels and regression theory
Neural Tangent Kernels for Axis-Aligned Tree Ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a closed-form neural tangent kernel for axis-aligned soft tree ensembles (AAA and AAI settings), shows non-oblivious trees reduce to oblivious ones with the same kernel, and uses multiple kernel learning to pick tree architectures.
Problem
Theoretical understanding of the learning behavior of axis-aligned tree ensembles is lacking because hard axis-aligned rules are discrete, and prior NTK analyses of soft trees assume oblique splits.
Approach
Impose the axis-aligned constraint on soft trees (fixing one nonzero feature per split at initialization, either throughout training (AAA) or only at initialization (AAI)), derive the limiting NTK in closed form, extend it to mixtures of architectures via kernel additivity, and show any non-oblivious axis-aligned ensemble maps to oblivious ones with the same NTK.
Key results
- Axis-aligned NTK loses rotational invariance, unlike the oblique soft-tree NTK, and its training trajectories converge to the analytical limit as the number of trees M grows (validated for M=16 vs 1024).
- Proposition 4.1 shows an asymmetric non-oblivious tree ensemble can be converted into oblivious tree ensembles with the same limiting NTK up to constant multiples, validated empirically with M=1024.
- On tic-tac-toe, AAA with MKL or the optimal kernel weighting outperforms AAA benchmark, RF and GBDT across alpha values, while AAI performs comparably between optimal and benchmark weightings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EasyMKL weights instead of uniform weights across AAA kernels | AAA · Tic-tac-toe | accuracy | Helped |
| Use outcome-informed interaction weights instead of uniform weights across AAA kernels | AAA · Tic-tac-toe | accuracy | Helped |
| Use EasyMKL weights instead of uniform weights across AAI kernels | AAI · Tic-tac-toe | accuracy | No clear effect |
| Use outcome-informed interaction weights instead of uniform weights across AAI kernels | AAI · Tic-tac-toe | accuracy | No clear effect |
| Apply the axis-aligned constraint throughout training rather than only at initialization | Soft-tree NTK SVM · Tic-tac-toe | accuracy | Helped |
| Use an axis-aligned-at-initialization constraint rather than oblique splitting | Soft-tree NTK SVM · Tic-tac-toe | accuracy | Helped |
| Use gradient-descent-based learning instead of greedy tree learning | AAA versus Random Forest and GBDT · Tic-tac-toe | accuracy | Helped |
| Replace a non-oblivious tree architecture with the converted oblivious-tree ensembles | Axis-aligned soft-tree ensemble · Synthetic training trajectories | training behavior | No clear effect |
Surprising findings
- Under the optimal hyperparameters, the axis-aligned gradient-trained model (AAA) outperforms the oblique kernel, suggesting stronger inductive bias helps on this dataset.
- AAA with uniform kernel weights (benchmark) still beats RF and GBDT on tic-tac-toe, suggesting gradient-based training matters beyond feature selection and softness.
Limitations
- Theory assumes features used at each split are predetermined; feature selection is handled only via MKL outside the analysis.
- Empirical validation is limited mainly to the tic-tac-toe dataset and small synthetic or toy experiments.
- Kernel analysis is for infinite-width limit with gradient flow and small learning rates, which may not match finite practical training.
Experimental setup
Datasets: tic-tac-toe diabetes 15 binary classification datasets from Fernández-Delgado et al. (2014)
Benchmarks: tic-tac-toe
Metrics: accuracy kernel value convergence output dynamics
Baselines: Oblique soft tree NTK Random Forest GBDT
Builds on: Neural Tangent Kernel Kanoh & Sugiyama soft tree NTK Neural Oblivious Decision Ensembles EasyMKL
Closest papers by meaning
- Neural Tangent Kernels Motivate Cross-Covariance Graphs in Neural Networks (ICML 2024 · similarity 0.87)
- GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data (ICLR 2024 · similarity 0.86)
- The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training (AAAI 2025 · similarity 0.85)
- GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent (AAAI 2024 · similarity 0.85)
- Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.85)
- Minimax Rates and Spectral Distillation for Tree Ensembles (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.