Research map / Architectures and efficiency / Attention and transformers
TabM: Advancing tabular deep learning with parameter-efficient ensembling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TabM, an MLP-based tabular model that imitates an ensemble of k=32 implicit submodels via parameter-efficient (BatchEnsemble-style) ensembling with improved initialization, and benchmarks it against tabular DL and GBDT models.
Problem
Tabular deep learning lacks a strong, consistent, and efficient go-to baseline; prior architectures (attention, retrieval-based) are often no better than MLPs and are costly on large datasets.
Approach
Apply BatchEnsemble-style parameter-efficient ensembling to tabular MLPs, training k implicit submodels simultaneously with shared weights and per-member adapters, initialized so that the first adapter alone gives diversity and the rest start as identity; ensemble-aware early stopping and tuning are applied to the collective prediction.
Key results
- TabM is reported as the best tabular DL model in the large-scale comparison, with the top performance rank among DL models and consistent improvements over plain MLP on most datasets.
- On large datasets, TabM with piecewise-linear embeddings (optimized) trains faster than FT-Transformer and TabR; e.g., on Maps Routing 2h vs 45.5h for FT-Transformer, with lower RMSE than XGBoost (0.1583 vs 0.1601).
- Individual submodels of TabM are overfitted and weak (best single submodel no better than a plain MLP), while their collective mean generalizes better; dead-neuron fraction drops from 0.29 (k=1) to 0.14 (k=32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train ensemble members collectively rather than as independent MLPs | MLP ensemble · tabular benchmarks | task performance | Helped |
| share weights between ensemble members | TabMpacked · tabular benchmarks | task performance | Helped |
| add the first input adapter (ablation) | TabMnaive · tabular benchmarks | task performance | Helped |
| keep only the first adapter and remove the remaining adapters (ablation) | TabMnaive · tabular benchmarks | task performance | Helped |
| initialize later multiplicative adapters deterministically with 1 (ablation) | TabMmini · tabular benchmarks | task performance | Helped |
| reuse the same training batches across submodels (ablation) | TabM · tabular benchmarks | task performance | Hurt |
| add non-linear feature embeddings (ablation) | TabMmini · tabular benchmarks | task performance | Helped |
| greedily prune submodels using validation performance (ablation) | TabM · tabular benchmarks | task performance | Hurt |
| increase the number of submodels from k=1 to k=32 (ablation) | TabMmini · 17 datasets | portion of dead neurons: 0.29 ± 0.17 -> 0.14 ± 0.09 | Helped |
Surprising findings
- Individually, even the best submodel of TabM is no better than a simple MLP, yet their average is much stronger.
- Plain MLP with BatchEnsemble already outperforms popular attention-based FT-Transformer, and a simple deep ensemble of MLPs is already better than FT-Transformer.
Limitations
- Uses plain MLP backbone; more advanced backbones with normalization layers were not explored.
- Hyperparameter k=32 is fixed heuristically; too large k can hurt performance.
- Efficiency benchmarks use tuned configurations that are not efficiency-aware, so measurements may overstate costs of heavier configurations.
Experimental setup
Datasets: 46 public tabular datasets incl. 28 random-split datasets from prior work TabReD benchmark (8 datasets with time-aware splits) Microsoft dataset Covertype Churn Modelling California Housing Higgs Small Adult
Benchmarks: 46-dataset tabular benchmark (38 random-split and 8 domain-aware splits) TabReD large datasets Weather and Maps Routing (full versions)
Metrics: accuracy ROC-AUC RMSE R^2-based relative performance vs MLP performance rank training time inference throughput parameter count
Baselines: MLP MLP×k deep ensemble FT-Transformer SAINT T2G-Former ExcelFormer TabR ModernNCA XGBoost LightGBM CatBoost
Builds on: BatchEnsemble Packed-Ensemble MiniEnsemble FT-Transformer piecewise-linear embeddings
Closest papers by meaning
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.91)
- TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.90)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.90)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.90)
- TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders (NeurIPS 2026 · similarity 0.89)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.