Research map / Large language models / Language, safety and interpretability
TabDPT: Scaling Tabular Foundation Models on Real Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TabDPT, a row-based tabular foundation model pre-trained on real OpenML data with retrieval-based in-context batches and column-masking SSL, showing competitive ICL performance on CC18/CTR23 and power-law scaling.
Problem
Building a tabular foundation model that generalizes to unseen datasets via in-context learning without per-dataset tuning, given that existing TFMs rely mostly on synthetic data and LLM-based approaches handle tables poorly.
Approach
Combine retrieval-based context selection during pre-training (aligning train with inference) with self-supervised column-masking targets on real tabular data, using a row-based transformer that treats each row as a token, and show real data with SSL beats synthetic-only pre-training and scales with parameters and cells.
Key results
- TabDPT reports the best average CC18 AUC (0.933) and accuracy (0.884) and CTR23 correlation (0.837) and R^2 (0.742), slightly above TabPFN v2 (0.932, 0.872, 0.835, 0.740).
- TabDPT (semi) averages 77.56% on 7 few-shot CC18 datasets, versus 76.76% for STUNT.
- Scaling experiments from 33K to 78M parameters and 52M to 2B cells fit a joint power law with alpha=0.42 and beta=0.39.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add self-supervised column-target training (ablation) | TabDPT trained on real data · CC18 and CTR23 | test performance | Helped |
| add retrieval when forming training contexts instead of random subsampling (ablation) | TabDPT · classification and regression | test accuracy | Helped |
| pre-train on real data with SSL rather than TabPFN synthetic data (ablation) | TabDPT · CC18 | test loss | Helped |
| use retrieval instead of subsampling at inference (ablation) | TabDPT · classification and regression | performance | Helped |
| use context size 2048 rather than 256 (ablation) | TabDPT · classification and regression | performance | Helped |
| increase pre-training data amount and model size | TabDPT · pre-training scaling experiments | excess loss: α = 0.42; β = 0.39 | Helped |
| remove feature-name embeddings (ablation) | TabDPT · preliminary experiments | performance | Helped |
| use robust preprocessing or soft-clipping variants (ablation) | TabDPT · pre-training | performance | No clear effect |
| use Robust Scaler or Power Transform preprocessing (ablation) | TabDPT · pre-training | training speed | Hurt |
| add class embeddings or class tokens with proto-network similarity (ablation) | TabDPT · especially real-data training | performance | Hurt |
| use alternative target embeddings for y_ctx (ablation) | TabDPT · large models with sufficient data | performance | No clear effect |
| add specialized NaN tokens or binary missingness features (ablation) | TabDPT · missing-value handling | performance | No clear effect |
Surprising findings
- Training on real data with SSL converges faster and generalizes better than purely synthetic TabPFN-generated data under the same compute budget.
- Without SSL, training on real data starts overfitting after around 50 epochs, whereas with SSL performance keeps improving past 500 epochs.
Limitations
- Designed for rectangular i.i.d. tables; does not model temporal dependencies, distribution shift, or hierarchical structure.
- Feature-name text embeddings caused overfitting, so free-form text is not integrated.
- Classical tree-based pipelines are about an order of magnitude faster for inference alone.
Experimental setup
Datasets: OpenML training collection (123 datasets) OpenML-CC18 OpenML-CTR23 STUNT few-shot protocol datasets
Benchmarks: OpenML-CC18 OpenML-CTR23 few-shot CC18 subsets (7 datasets)
Metrics: AUC accuracy correlation R^2 Elo Glicko2 win rate
Baselines: TabPFN v2 TabPFN (kNN) TabPFN Tabula-8B TabM TabR MLP-PLR MLP XGBoost LightGBM CatBoost STUNT
Builds on: TabPFN TabPFN v2 STUNT Schedule-Free optimizer FAISS
Closest papers by meaning
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.93)
- Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.93)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.92)
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.92)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.91)
- Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.