Research map / Large language models / Language, safety and interpretability

Understanding the Surprising Generalization Properties of Tabular Foundation Models

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that tabular in-context foundation models (TabDPT-style) pre-trained on a single real table transfer surprisingly well across domains, that feature count and task count drive generalization, that column-level preprocessing helps while dataset-level filtering does not, and that ICL generalization is better explained by learned retrieval-and-aggregation than by a Bayesian prior.

Problem

Tabular foundation models are assumed to need very diverse pre-training corpora to generalize; it is unclear which properties of pre-training data drive in-context generalization to unseen tabular tasks and what mechanism underlies it.

Approach

Pre-training on a single real table with a self-supervised randomized column-as-target objective yields broad transfer; generalization is governed by the number of distinct tasks (feature-target configurations), which feature count drives, and TFMs behave like learned retrieval-and-aggregation procedures rather than parametric Bayesian predictors.

Key results

• Column-level preprocessing (NaN + Spearman dedup + mf-5) improves CTR-23 IQM R^2 from 0.8145 to 0.8251 and average from 0.9039 to 0.9091, while dataset-level deduplication gives no gain (avg 0.9038).
• Filtering to datasets with at least 30 features degrades performance (avg 0.8927 vs 0.9039 baseline).
• Across 88 single-table pre-trained models, in-context retrieval AUC correlates with CC-18 base AUC at Pearson 0.89; attention similarity correlates with held-out AUC at Spearman 0.537.
• On TabArena-Lite, TabDPT 4k+pp reaches Elo 1322.8, above CatBoost (1320.8) and the 4k baseline (1274.3).

Surprising findings

• Pre-training on a single real table (e.g., vectorized MNIST or Colleges) yields strong transfer to unrelated benchmarks.
• Pairing a strong single-table dataset with a weak one does not hurt and can slightly improve performance; exact deduplication of the corpus has no measurable effect.

Limitations

• Only real-world pre-training data was studied; findings may not extend to purely synthetic pre-training.
• Only row-based attention architectures (TabDPT/TabPFNv1 style) were analyzed, not cell-based ones.
• Retrieval mechanism explanation is not proven; other mechanisms beyond learned retrieval may contribute.

Experimental setup

Datasets: MNIST (vectorized) Colleges OpenML 1,732-dataset corpus OpenML CC-18 OpenML CTR-23 TabArena

Benchmarks: OpenML CC-18 OpenML CTR-23 TabArena-Lite

Metrics: AUC accuracy R^2 correlation IQM Elo win rate Spearman correlation

Baselines: XGBoost Random Forest Linear Model TabPFNv2 TabICL CatBoost TabDPT

Builds on: TabPFN TabDPT TabArena k-nearest neighbours XGBoost

Closest papers by meaning

• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.93)
• Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.91)
• UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science (ICLR 2024 · similarity 0.90)
• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.90)
• Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.90)
• Enhancing Tabular Learners with Context-Aware Semantic Embeddings (NeurIPS 2026 · similarity 0.89)