Research map / Label-efficient and robust learning / Recognition and long-tail
DOFEN: Deep Oblivious Forest ENsemble
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOFEN, a DNN that randomly combines per-column soft conditions into relaxed oblivious decision trees and ensembles them in a two-level forest scheme, reaching SOTA among DNNs on the Tabular Benchmark.
Problem
DNNs still lag behind gradient-boosted decision trees on tabular data; the paper aims to bring tree-like inductive biases and ensembling into a neural architecture to close this gap.
Approach
Replaces the non-differentiable ODT learning with randomly selected column conditions scored by sub-networks (soft conditions), forming relaxed ODTs (rODTs) that are sampled into rODT forests with learned weights and bagged, with the random combinations fixed once at construction.
Key results
- On medium-sized Tabular Benchmark classification, DOFEN default reaches 0.7725 mean accuracy, close to XGBoost 0.7717 and CatBoost 0.7777 at default settings (table), while being the best DNN in the comparison.
- Regression: DOFEN default R2 0.6611 vs Trompt 0.6541 and XGBoost 0.6719 (default); with optimal hyperparameters DOFEN 0.6882 vs XGBoost 0.6985.
- Ablation: removing the forest ensemble drops classification accuracy from 0.7725 to 0.7362 and removing condition shuffling gives 0.7709.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove condition shuffling (ablation) | DOFEN · Medium classification | accuracy: 0.7725 -> 0.7709 | Hurt |
| remove condition shuffling (ablation) | DOFEN · Medium regression | R-squared: 0.6605 -> 0.6439 | Hurt |
| remove forest ensemble (ablation) | DOFEN · Medium classification | accuracy: 0.7725 -> 0.7362 | Hurt |
| remove forest ensemble (ablation) | DOFEN · Medium regression | R-squared: 0.6605 -> 0.3238 | Hurt |
| remove condition shuffling and forest ensemble (ablation) | DOFEN · Medium classification | accuracy: 0.7725 -> 0.7323 | Hurt |
| remove condition shuffling and forest ensemble (ablation) | DOFEN · Medium regression | R-squared: 0.6605 -> 0.5441 | Hurt |
| increase forest count from 1 to 100 | DOFEN · jannis | accuracy: 0.7382 -> 0.7808 | Helped |
| increase forest count from 1 to 100 | DOFEN · road-safety | accuracy: 0.7517 -> 0.7732 | Helped |
| increase forest count from 1 to 100 | DOFEN · delays-zurich | R-squared: 0.0054 -> 0.0268 | Helped |
| increase forest count from 1 to 100 | DOFEN · abalone | R-squared: 0.5469 -> 0.5868 | Helped |
Surprising findings
- DOFEN has the lowest FLOPs among DNNs yet the longest inference time, attributed to inefficient group convolution implementations.
- Removing the forest ensemble collapses regression R2 to 0.3238 mainly due to an outlier dataset, not uniformly.
Limitations
- Inference time is relatively long despite low FLOPs, due to group convolution implementation.
- Randomization leads to slower convergence, requiring more training epochs.
- Heterogeneous (categorical) features show weaker performance relative to numerical-only datasets.
Experimental setup
Datasets: Tabular Benchmark (73 datasets) mushroom red wine quality white wine quality FT-Transformer datasets GRANDE datasets
Benchmarks: Tabular Benchmark
Metrics: accuracy R-squared score
Baselines: XGBoost CatBoost LightGBM RandomForest GradientBoostingTree HGBT MLP ResNet SAINT FT-Transformer NODE Trompt
Builds on: oblivious decision tree NODE GRANDE Trompt
Closest papers by meaning
- Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (ICLR 2020 · similarity 0.88)
- GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data (ICLR 2024 · similarity 0.83)
- Sensitivity Verification for Additive Decision Tree Ensembles (ICLR 2025 · similarity 0.81)
- Why do tree-based models still outperform deep learning on typical tabular data? (NeurIPS 2022 · similarity 0.81)
- Neural Tangent Kernels for Axis-Aligned Tree Ensembles (ICML 2024 · similarity 0.80)
- Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again (NeurIPS 2026 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.