Research map / Learning theory and algorithms / Prediction and decision losses
When Do Neural Nets Outperform Boosted Trees on Tabular Data?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TabPFN instead of other evaluated algorithms | Tabular classification · 98 OpenML classification datasets | Log loss rank | Helped |
| Use TabPFN* rather than the other evaluated algorithms | Tabular classification · 57 OpenML datasets with size at most 1250 | Mean normalized accuracy | Helped |
| Choose a GBDT rather than a neural net | Tabular classification · 176 OpenML classification datasets | Average performance | Mixed |
| Lightly tune CatBoost rather than choosing between default GBDT and NN algorithms | Tabular classification · About one-third of the 176 datasets | Normalized log loss | Helped |
| Use GBDTs on datasets with skewed, heavy-tailed, or high-variance feature distributions | GBDTs versus neural nets · Irregular tabular datasets | Normalized log loss | Helped |
| Use GBDTs rather than neural nets as dataset size grows | GBDTs versus neural nets · Larger tabular datasets | Relative performance | Helped |
| Use GBDTs when the dataset-size-to-feature-count ratio is high | GBDTs · Tabular classification datasets with high size-to-feature ratio | Relative performance | Helped |
| Use simple baselines rather than more complex models when they reach comparable performance | Tabular classification · A surprisingly high fraction of the studied datasets | Accuracy | No clear effect |
| Randomly sample 3000 training datapoints for TabPFN on larger datasets | TabPFN · Larger tabular datasets | Accuracy | Helped |
| Use TabPFN rather than CatBoost on small datasets | Tabular classification · 57 OpenML datasets with size at most 1250 | Mean normalized accuracy: 0.84 vs. 0.85 | No clear effect |
Closest papers by meaning
- Better by default: Strong pre-tuned MLPs and boosted trees on tabular data (NeurIPS 2024 · similarity 0.89)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.87)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.87)
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.86)
- NRGBoost: Energy-Based Generative Boosted Trees (ICLR 2025 · similarity 0.86)
- TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.