Research map / Architectures and efficiency / RNNs and normalization
Linear Mode Connectivity in Differentiable Tree Ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows linear mode connectivity can be achieved for soft tree ensembles by accounting for subtree flip and splitting order invariances, and that decision-list-based architectures achieve LMC with only permutation invariance.
Problem
Whether linear mode connectivity (LMC), established for neural networks via permutation invariance, can be achieved for differentiable soft tree ensembles, which have architecture-specific invariances.
Approach
Identifies subtree flip and splitting order invariances inherent to soft tree architectures and matches models by searching over them together with tree permutations (via LAP and weighting by leaf-coverage); alternatively designs a modified decision-list architecture with an empty leaf so that only tree permutation invariance remains.
Key results
- For oblivious trees with D=2 and M=256, the averaged test accuracy barrier under weight matching drops from 7.881 (naive) and 0.918 (permutation only) to 0.348 when all invariances are considered.
- Modified decision lists achieve smaller barriers than original decision lists under permutation-only matching (e.g., test WM barrier 0.468 vs 0.436 is comparable, but AM test barrier 2.114 vs 7.452).
- Non-oblivious tree barriers under AM test drop from 8.631 (perm) to 0.943 (ours) on average over 16 datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add subtree flip matching to tree permutation matching (ablation) | Non-Oblivious Tree with WM, D=2 · Tabular-Benchmark (16 datasets), test | accuracy barrier: 0.811 ± 0.333 -> 0.455 ± 0.105 | Helped |
| Add subtree flip and splitting order matching to tree permutation matching (ablation) | Oblivious Tree with WM, D=2 · Tabular-Benchmark (16 datasets), test | accuracy barrier: 0.919 ± 0.093 -> 0.348 ± 0.172 | Helped |
| Add subtree flip and splitting order matching to tree permutation matching (ablation) | Oblivious Tree with AM, D=2 · Tabular-Benchmark (16 datasets), test | accuracy barrier: 10.349 ± 0.476 -> 0.395 ± 0.185 | Helped |
| Add subtree flip matching to tree permutation matching (ablation) | Non-Oblivious Tree with AM, D=2 · Tabular-Benchmark (16 datasets), test | accuracy barrier: 8.631 ± 1.444 -> 0.943 ± 0.435 | Helped |
| Replace an original decision list with a modified decision list that fixes one terminal leaf to zero (ablation) | Decision List with WM · Tabular-Benchmark (16 datasets), test, D=2 | accuracy barrier: 0.436 ± 0.120 -> 0.468 ± 0.150 | Mixed |
| Replace an original decision list with a modified decision list that fixes one terminal leaf to zero (ablation) | Decision List with AM · Tabular-Benchmark (16 datasets), test, D=2 | accuracy barrier: 7.452 ± 1.840 -> 2.114 ± 0.243 | Helped |
| Use only tree permutation matching instead of matching additional tree invariances (ablation) | Non-Oblivious Tree with WM, D=2 · Tabular-Benchmark (16 datasets), test | accuracy barrier: 0.455 ± 0.105 -> 0.811 ± 0.333 | Hurt |
| Use only tree permutation matching instead of matching additional tree invariances (ablation) | Oblivious Tree with WM, D=2 · Tabular-Benchmark (16 datasets), test | accuracy barrier: 0.348 ± 0.172 -> 0.919 ± 0.093 | Hurt |
| Increase tree depth from D=1 to D=3 | Non-Oblivious Tree with Ours matching · Tabular-Benchmark (16 datasets), test | accuracy barrier: 0.181 ± 0.078 -> 0.740 ± 0.158 | Hurt |
| Interpolate parameters of models trained on complementary split datasets | Oblivious Tree with AM, D=2 and M=256 · Tabular-Benchmark split-data experiments, test | test accuracy | Helped |
Surprising findings
- Interpolating models trained on split datasets with different class ratios can yield performance improvements, sometimes approaching full-data training performance.
- Deeper trees have larger barriers even after accounting for additional invariances, so deep models are deprioritized for merging.
Limitations
- Exhaustive search over additional invariances becomes impractical for deep trees.
- Modified decision lists still have larger barriers than oblivious trees with additional invariances under AM/WM.
Experimental setup
Datasets: Tabular-Benchmark (16 datasets, e.g., MiniBooNE, Higgs, covertype, jannis) MNIST
Benchmarks: Tabular-Benchmark (16 datasets) MNIST
Metrics: accuracy barrier test accuracy train accuracy
Baselines: Naive interpolation Tree permutation only MLP with permutation matching
Builds on: linear mode connectivity git re-basin weight matching soft tree ensembles neural tangent kernel
Closest papers by meaning
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.87)
- Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods (ICLR 2025 · similarity 0.87)
- Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries (ICML 2025 · similarity 0.86)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.85)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.85)
- Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.