Research map / Vision-language and multimodal / Cross-modal learning

Position: Why Tabular Foundation Models Should Be a Research Priority

ICML 2024 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper arguing that tabular foundation models (termed Large Tabular Models, LTMs) are a neglected research priority and outlining their desiderata, current state, applications, challenges, and comparison with LLMs.

Problem

Tabular data is the dominant modality in many fields but receives little foundation-model research attention compared with text and image, and existing tabular ML lags in scale and generality.

Approach

Proposes the Large Tabular Model (LTM) as a foundation model for tabular data, defines four model desiderata (mixed-type columns, cross-dataset modelling, textual context, column-order invariance/equivariance), argues LLMs are inefficient for numerical tabular generation, and maps applications in responsible AI and science.

Key results

• TabLib is cited as a metadataset of 627 million tables and 827 billion context tokens.
• Keyword counts of accepted papers (ICML 2023, NeurIPS 2023, ICLR 2024) show tables/tabular papers heavily underrepresented relative to LLM, image, and graph research.
• Argues that developing SOTA LTMs is within reach of many researchers, since tabular datasets are typically small (e.g. largest benchmark datasets ~50,000 points).

Surprising findings

• Argues that LLMs are poor at modelling continuous tabular variables because numbers are tokenized into bins, making autoregressive Gaussian sampling inefficient.
• Notes that TabPFN, trained only on synthetic data, claims to outperform XGBoost on real datasets.

Limitations

• No experiments; claims rest on literature review and keyword counts.
• Keyword-based estimates of modality representation are rough and the authors acknowledge the comparison is between apples and pears.

Experimental setup

Datasets: TabLib WebTables WikiTables

Baselines: XGBoost SAINT LLMs for tabular generation

Builds on: foundation model definition (Bommasani et al.) TabPFN GReaT

Closest papers by meaning

• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.88)
• Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.88)
• Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.88)
• TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.86)
• A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.86)
• TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments (ICML 2025 · similarity 0.85)