Research map / Vision-language and multimodal / Human and embodied modeling
TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TabSTAR, a tabular foundation model that unfreezes a pretrained text encoder (e5-small-v2) and uses target-aware tokens to handle free-text tabular features without dataset-specific parameters.
Problem
Deep tabular models and tabular foundation models handle free-text and high-cardinality features poorly, usually relying on static, target-agnostic text embeddings from frozen encoders.
Approach
Unfreezes the upper half of a pretrained text encoder and feeds all possible target values as input tokens (target-aware tokens), so a shared prediction head works across classes and datasets with no dataset-specific parameters; verbalizes numerical features with z-scores plus quantile-bin text.
Key results
- On classification with up to 10K examples, TabSTAR reaches a normalized score of 0.83, ahead of TabM-Tuned (0.73), CatBoost-Tuned (0.69), and TabPFN-v2 (0.67).
- With unlimited training size, TabSTAR-Unlimited achieves 0.84 versus 0.79 for TabM-Tuned-Unlimited in classification.
- In regression, TabSTAR is competitive at 10K but does not reach SOTA; GBDTs remain stronger.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TabSTAR instead of the strongest reported baseline | classification models · Classification, 10K | normalized score: 0.73 -> 0.83 | Helped |
| use TabSTAR-Unlimited instead of TabM-Tuned-Unlimited | classification models · Classification, Unlimited | normalized score: 0.79 -> 0.84 | Helped |
| add numerical information to feature verbalization rather than using Name only (ablation) | TabSTAR · Classification | normalized score: 0.386 ± 0.095 -> 0.593 ± 0.097 | Helped |
| add numerical information to feature verbalization rather than using Name only (ablation) | TabSTAR · Regression | normalized score: 0.386 ± 0.081 -> 0.596 ± 0.079 | Helped |
| include quantile information in numerical feature verbalization (ablation) | TabSTAR · Classification | normalized score: 0.544 ± 0.093 -> 0.593 ± 0.097 | Helped |
| include quantile information in numerical feature verbalization (ablation) | TabSTAR · Regression | normalized score: 0.584 ± 0.076 -> 0.596 ± 0.079 | Helped |
| unfreeze encoder layers instead of using static embeddings (ablation) | TabSTAR · downstream tasks | normalized score | Helped |
| unfreeze 6 encoder layers rather than 9 (ablation) | TabSTAR · downstream tasks | normalized score | Helped |
| pretrain before downstream finetuning rather than finetune from scratch (ablation) | TabSTAR · downstream tasks | performance | Helped |
| increase the number of pretraining datasets (ablation) | TabSTAR · classification and regression downstream tasks | AUROC / R² | Helped |
| use attention-based fusion instead of concatenation or multiplication (ablation) | TabSTAR · pretraining evaluation | performance | Helped |
| use 6 interaction layers rather than 3 (ablation) | TabSTAR · pretraining evaluation | performance | Helped |
Surprising findings
- TabPFN-v2 significantly underperforms GBDTs on regression with text features despite strong results on non-textual data.
- Fine-tuning with differential (lower) learning rate for the text encoder hurt performance.
Limitations
- Regression results lag behind GBDTs and other TFMs in the 10K condition.
- Training is slower than untuned GBDTs and ICL-based TFMs; memory bottlenecks on datasets with many features.
- Not extensively evaluated in few-shot settings or purely numerical datasets.
Experimental setup
Datasets: AutoML Multimodal Benchmark CARTE benchmark OpenML-CTR23 TabZilla AMLB Kaggle OpenML
Benchmarks: AutoML Multimodal Benchmark CARTE benchmark free-text and high-cardinality feature benchmark (50 datasets)
Metrics: AUROC R^2 normalized score
Baselines: Random Forest LightGBM XGBoost CatBoost RealMLP TabM TabPFN-v2 TabICL TabDPT CARTE
Builds on: TabPFN-v2 TP-BERTa CARTE SAINT LoRA
Closest papers by meaning
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.91)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.90)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.90)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.90)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.90)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.