Research map / Architectures and efficiency / Pruning and efficient tuning
Better by default: Strong pre-tuned MLPs and boosted trees on tabular data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RealMLP, an improved MLP with a bag of tricks, plus meta-tuned default hyperparameters for GBDTs and RealMLP, showing that trying tuned defaults is a favorable time-accuracy tradeoff versus per-dataset HPO on tabular data.
Problem
Deep learning methods for tabular classification and regression are often much slower and require extensive per-dataset hyperparameter tuning, while GBDTs dominate; the cost of HPO hinders adoption of new methods.
Approach
Improve MLPs with a bag of tricks (robust scaling with smooth clipping, PBLD numerical embeddings, scaling layer, NTP linear layers, parametric activations, multi-cycle coslog learning-rate schedule, data-dependent initialization) and meta-tune default hyperparameters for MLPs and GBDTs on a separate meta-train benchmark, evaluated on a disjoint meta-test benchmark.
Key results
- RealMLP-TD is competitive with GBDTs on benchmark scores and outperforms other comparably fast NN baselines, with RealMLP-TD beating TabR-S-D on many benchmarks.
- RealTabR-D performs better than RealMLP-TD on four of six benchmarks, especially on all regression benchmarks, showing the bag of tricks transfers to TabR.
- Best-TD (trying tuned default algorithms) is faster on average than 50-step HPO on RealMLP or GBDTs while remaining competitive with the best HPO model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PBLD numerical embeddings rather than PLR embeddings (ablation) | RealMLP-TD · Meta-train regression | SGM error: 19.0 [13.7, 24.5] error increase with PLR | Helped |
| Add numerical embeddings rather than using none (ablation) | RealMLP-TD · Meta-train regression | SGM error: 20.6 [19.4, 21.8] error increase with no embeddings | Helped |
| Set Adam β₂ to 0.95 rather than 0.999 (ablation) | RealMLP-TD · Meta-train regression | SGM error: 22.8 [21.3, 24.4] error increase with β₂=0.999 | Helped |
| Use a multi-cycle rather than constant learning-rate schedule (ablation) | RealMLP-TD · Meta-train regression | SGM error: 13.5 [11.9, 15.0] error increase with constant schedule | Helped |
| Add label smoothing (ablation) | RealMLP-TD · Meta-train classification | SGM error: 1.8 [1.2, 2.5] error increase without label smoothing | Helped |
| Add the learnable scaling layer (ablation) | RealMLP-TD · Meta-train classification | SGM error: 1.4 [0.7, 2.1] error increase without scaling | Helped |
| Use parametric rather than non-parametric activations (ablation) | RealMLP-TD · Meta-train regression | SGM error: 4.8 [3.4, 6.2] error increase with non-parametric activation | Helped |
| Use Mish rather than SELU (ablation) | RealMLP-TD · Meta-train regression | SGM error: 2.3 [1.2, 3.6] error increase with SELU | Helped |
| Schedule dropout with flat_cos rather than using constant dropout (ablation) | RealMLP-TD · Meta-train regression | SGM error: 3.6 [3.0, 4.2] error increase with constant dropout | Helped |
| Schedule weight decay with flat_cos rather than using constant weight decay (ablation) | RealMLP-TD · Meta-train regression | SGM error: 3.1 [1.7, 4.4] error increase with constant weight decay | Helped |
| Use data-dependent rather than standard-normal/zero-bias initialization (ablation) | RealMLP-TD · Meta-train classification | SGM error: 0.9 [0.1, 1.8] error increase without data-dependent initialization | Helped |
| Use robust scaling and smooth clipping rather than the alternative numerical preprocessing methods (ablation) | RealMLP-TD-S · Meta-train classification and regression | SGM error | Helped |
Surprising findings
- Simply trying multiple tuned default algorithms gives a better time-accuracy tradeoff than naive single-algorithm HPO, contrary to claims favoring CatBoost-HPO over NNs.
- Label smoothing improves classification error but can be detrimental for AUROC, since cross-entropy with label smoothing is not a proper scoring rule.
Limitations
- Unclear generalization to very small datasets, distribution shifts, datasets with missing numerical values, and metrics like log-loss.
- Only a single training-validation split per train-test split is used, so HPO may overfit the validation set more easily; GBDT depth and tree limits are on the low side of the literature.
- Non-architectural aspects are not equalized, so the work is not a pure architecture comparison.
Experimental setup
Datasets: UCI Repository (meta-train) AutoML Benchmark (meta-test) OpenML-CTR23 (meta-test) Grinsztajn et al. 2022 benchmark
Benchmarks: meta-train benchmark meta-test benchmark Grinsztajn et al. 2022 benchmark
Metrics: shifted geometric mean error (SGM) classification error 1-AUROC nRMSE mean normalized error win-rate training time
Baselines: XGBoost LightGBM CatBoost Random forest MLP ResNet FT-Transformer MLP-PLR TabR XGB-PBB-D
Builds on: TabR MLP-PLR XGBoost LightGBM CatBoost
Closest papers by meaning
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.86)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.85)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.85)
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.85)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.85)
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.