Research map / Learning theory and algorithms / Prediction and decision losses
Invariant Random Forest: Tree-Based Model Solution for OOD Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Invariant Decision Tree (IDT) and Invariant Random Forest (IRF), which add a penalty on how much a split's changing-rate ratio varies across environments, to improve OOD generalization of tree models.
Problem
Standard decision trees and random forests choose splits that maximize average performance and ignore unstable behavior of splits across environments, so they generalize poorly under distribution shift (OOD).
Approach
Derives an invariant (ratio of positive and negative changing rates after a split) that holds across environments for stable splitting variables, and adds a penalty on its variance across training environments to the Gini-based split objective; the forest omits random feature subsampling and bootstraps per environment.
Key results
- On synthetic classification with d=20, IRF (lambda=10) reaches 57.42% accuracy vs 40.08% for RF and 48.28% for XGBoost.
- On real regression tasks in Scenario 1, IRF reduces MSE relative to RF by up to about 15% on some derived datasets (e.g., Energy1-4 at 0.892 with lambda=10).
- XGBoost performed worse than RF in all OOD tasks in the paper, which the authors attribute to bagging being better than boosting under OOD settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IRF with λ = 10 instead of RF | Synthetic classification d=20 · Synthetic classification d=20 | accuracy: 40.08 (0.79) -> 57.42 (0.47) | Helped |
| Increase λ from 1 to 10 (ablation) | IRF · Synthetic classification d=20 | accuracy: 52.56 (1.02) -> 57.42 (0.47) | Helped |
| Increase λ from 1 to 10 (ablation) | IRF · Synthetic regression d=20 | MSE: 0.990 (0.009) -> 0.684 (0.011) | Helped |
| Increase λ from 1 to 10 (ablation) | IRF · Synthetic classification d=2 | accuracy: 50.24 (0.96) -> 51.06 (0.62) | Mixed |
| Increase λ from 1 to 10 (ablation) | IRF · Synthetic regression d=2 | MSE: 0.982 (0.014) -> 0.796 (0.014) | Helped |
| Increase λ from 1 to 10 (ablation) | IRF · Financial classification, Scenario 1 | accuracy: 55.30 (1.27) -> 56.84 (1.47) | Helped |
| Increase λ from 1 to 10 (ablation) | IRF · Technical classification, Scenario 1 | accuracy: 49.36 (0.45) -> 50.65 (0.75) | Helped |
| Increase λ from 1 to 10 (ablation) | IRF · Asset Pricing classification, Scenario 1 | accuracy: 51.96 (0.39) -> 51.85 (0.42) | Mixed |
| Use IRF instead of RF | IRF · Financial classification, Scenario 3 | accuracy: 54.50 (1.33) -> 54.27 (1.22) | Hurt |
| Use IRF instead of RF | IRF · Energy2-1 regression, Scenario 2 | MSE: 1.000 -> 1.006 (0.050) | Hurt |
| Use IRF instead of RF | IRF · Energy1-1 regression, Scenario 1 | MSE: 1.000 -> 1.009 (0.007) | Hurt |
| Increase λ from 1 to 10 (ablation) | IRF · Energy1-1 regression, Scenario 1 | MSE: 1.009 (0.007) -> 1.027 (0.024) | Mixed |
Surprising findings
- XGBoost consistently underperformed RF across all OOD generalization tasks in the paper.
Limitations
- Choice of penalty weight lambda is unclear without a validation set (Scenario 1).
- Assumes meta-information or environment partitions exist; real-data environments are hand-made via Decorr in some scenarios.
- Some real-data gains are small or within standard deviations, and IRF is not uniformly best (e.g., Asset Pricing).
Experimental setup
Datasets: Synthetic classification (generated) Synthetic regression (generated) Financial indicator dataset (Kaggle) US stock technical indicators (Kaggle) Asset Pricing Energy datasets (Appliances energy, Tetuan city power consumption) Air quality PRSA dataset
Benchmarks: Synthetic classification Synthetic regression Financial Technical Asset Pricing Energy1-i/Energy2-i Air1-i/Air2-i
Metrics: accuracy MSE (relative to RF)
Baselines: Random Forest XGBoost Linear regression IRM REx
Builds on: Random Forest REx IRM Time Robust Tree
Closest papers by meaning
- Hierarchical Shrinkage: Improving the accuracy and interpretability of tree-based models. (ICML 2022 · similarity 0.88)
- Learning Small Decision Trees with Few Outliers: A Parameterized Perspective (AAAI 2024 · similarity 0.87)
- Towards Better Decision Forests: Forest Alternating Optimization (CVPR 2023 · similarity 0.87)
- Symbolic Regression Enhanced Decision Trees for Classification Tasks (AAAI 2024 · similarity 0.85)
- Generative Model for Decision Trees (AAAI 2024 · similarity 0.85)
- Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithm (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.