Research map / Large language models / AI and society
TabArena: A Living Benchmark for Machine Learning on Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TabArena, a continuously maintained living benchmark for IID tabular classification and regression, with 51 curated datasets, 16 models with tuning and ensembling, and a public leaderboard.
Problem
Existing tabular benchmarks are static, contain outdated or flawed datasets, use weak evaluation protocols, and are not maintained, so they do not reflect state-of-the-art performance.
Approach
Treat the benchmark as versioned, maintained software with manual dataset curation, protocols for models, datasets and evaluation (8-fold inner CV, post-hoc ensembling of 200 random configurations), Elo-based aggregation, and a public leaderboard that accepts new submissions.
Key results
- RealMLP (T+E) achieves the top Elo of 1564 on TabArena-v0.1, with TabM (T+E) at 1541 and LightGBM (T+E) at 1527.
- TabPFNv2 wins most often (11.0 average wins) and has the best harmonic mean rank (3.1) across the full leaderboard, with lower improvability on small datasets.
- Ensembling across all models yields a simulated ensemble that outperforms every individual model and AutoGluon, and models overfitting validation data (ModernNCA, RealMLP) are over-selected in the ensemble.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add weighted post-hoc ensembling of tuned hyperparameter configurations | TabM · TabArena-v0.1 | Elo: 1447${}_{-52,+73}$ -> 1541${}_{-61,+74}$ | Helped |
| add weighted post-hoc ensembling of tuned hyperparameter configurations | RealMLP · TabArena-v0.1 | Elo: 1342${}_{-45,+43}$ -> 1564${}_{-57,+73}$ | Helped |
| add weighted post-hoc ensembling of tuned hyperparameter configurations | LightGBM · TabArena-v0.1 | Elo: 1446${}_{-37,+39}$ -> 1527${}_{-45,+49}$ | Helped |
| add weighted post-hoc ensembling of tuned hyperparameter configurations | CatBoost · TabArena-v0.1 | Elo: 1469${}_{-49,+56}$ -> 1482${}_{-55,+57}$ | Helped |
| use holdout validation instead of cross-validation for model selection (ablation) | tuned and ensembled models · TabArena-v0.1 | predictive performance | Hurt |
| increase ModernNCA tuning beyond 25 configurations (ablation) | ModernNCA · TabArena-v0.1 | Elo | Hurt |
| train TabM and ModernNCA on CPU rather than GPU with the 1-hour time limit (ablation) | TabM and ModernNCA · TabArena-v0.1 | predictive performance: approximately 16% of runs are early stopped on CPU; less than 0.1% on GPU | Hurt |
| train RealMLP on CPU rather than GPU with the 1-hour time limit (ablation) | RealMLP · TabArena-v0.1 | predictive performance: less than 0.1% of runs are early stopped due to the time limit | No clear effect |
| use one outer fold in TabArena-Lite instead of the full evaluation design (ablation) | TabArena-Lite · TabArena-Lite | model performance assessment | No clear effect |
| ensemble across all benchmarked models | TabArena ensemble · TabArena-v0.1 | predictive performance | Helped |
| use TabPFNv2 for datasets within its constraints | tabular foundation models · TabPFNv2-compatible datasets | predictive performance | Helped |
| use TabPFNv2 with tuning and post-hoc ensembling rather than AutoGluon | TabPFNv2 · TabPFNv2-compatible datasets | predictive performance | Helped |
Surprising findings
- Holdout validation strongly underestimates all models and biases results toward models that already use ensembling.
- Top three leaderboard models (TabM, LightGBM, RealMLP) would be worse than CatBoost without post-hoc ensembling.
Limitations
- Fixed set of 200 random hyperparameter configurations prevents studying advanced HPO and variance of random search.
- Per-configuration time limit of 1 hour makes results hardware-dependent in edge cases.
- Strict selection criteria yield only 51 datasets, limiting statistical power and diversity; no feature engineering evaluated.
Experimental setup
Datasets: PMLB(Mini) OpenML-CC18 OpenML-CTR23 AMLB TabZilla TabRepo TALENT PyTabKit
Benchmarks: TabArena-v0.1
Metrics: Elo normalized score average rank harmonic mean rank improvability ROC AUC log-loss RMSE
Baselines: AutoGluon CatBoost LightGBM XGBoost RandomForest ExtraTrees TabM RealMLP ModernNCA TabPFNv2 TabICL TabDPT
Builds on: AutoGluon TabRepo AutoML Benchmark TALENT
Closest papers by meaning
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.90)
- TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks (NeurIPS 2026 · similarity 0.90)
- TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments (ICML 2025 · similarity 0.90)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.90)
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.89)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.