Research map / Architectures and efficiency / Attention and transformers
TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TabICL, a scalable tabular foundation model for classification that uses column-then-row attention to embed rows before dataset-wise in-context learning, pretrained on synthetic data up to 60K samples.
Problem
In-context learning tabular foundation models like TabPFNv2 have quadratic and column-row attention costs that make them impractical for large tables (tens to hundreds of thousands of samples).
Approach
A two-stage architecture: a distribution-aware set-transformer column embedding plus a row-wise transformer with RoPE and [CLS] tokens collapses each row into a fixed-dimensional embedding, then a transformer performs ICL over row embeddings, giving O(m^2 n + n^2) complexity instead of TabPFNv2's O(m^2 n + n^2 m). Also adds tree-based SCM priors, curriculum pretraining up to 60K samples, and hiera
Key results
- TabICL is on par with TabPFNv2 on TALENT classification datasets (difference not statistically significant) while being 1.5x faster on small datasets and 3-10x faster on large datasets.
- On 53 TALENT datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost.
- TabICL handles 100K samples and 500 features with about 5 GB GPU memory, and 500K samples with under 14 GB GPU memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TabICL instead of TabPFNv2 | Tabular classification · 10,000 samples, 100 features | runtime: roughly 20 seconds versus 1 minute 40 seconds | Helped |
| use TabICL instead of TabPFNv2 | Tabular classification · 1,000 samples, 10 features | runtime: 1 second compared to 2 seconds | Helped |
| add tree-based SCMs to synthetic pretraining (ablation) | TabICL · 200 classification datasets | relative accuracy | Helped |
| increase pretraining dataset sizes from fixed 1,024 to randomly sampled 1K–40K (ablation) | TabICL · 200 classification datasets | average rank: 11.4 to 7.46 | Helped |
| add pretraining on 40K–60K samples while training only the ICL component (ablation) | TabICL · 200 classification datasets | average rank: 7.46 to 6.95 | Helped |
| use curriculum learning with progressively larger pretraining datasets (ablation) | TabICL · small classification datasets | accuracy | Hurt |
Surprising findings
- Curriculum learning raised average rank from 11.4 after stage 1 to 6.95 after stage 3, though some small datasets dropped slightly.
- TabICL's column permutation invariance is violated by RoPE, though ensembling over column permutations approximately restores it.
Limitations
- Slow inference speed typical of foundation models; currently limited to classification.
- Violates column-permutation invariance due to RoPE.
- Evaluation inherits TALENT benchmark limitations (holdout tuning of baselines, mean imputation, no ensembling for other models).
Experimental setup
Datasets: TALENT benchmark (evaluation) synthetic SCM and tree-based SCM prior datasets
Benchmarks: TALENT (200 classification datasets, 171 with at most 10 classes) TALENT datasets with more than 10 classes (12) TALENT datasets with over 10K samples (53)
Metrics: accuracy relative accuracy over tuned MLP log loss AUC training+inference time rank
Baselines: TabPFNv2 CatBoost TabR ModernNCA RealMLP TabM TabPFN (v1) LoCalPFN TuneTables
Builds on: TabPFN TabPFNv2 Set Transformer RoFormer rotary positional embedding XGBoost
Closest papers by meaning
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.93)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.91)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.91)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.91)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.89)
- TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.