Research map / Graph learning / Knowledge and temporal graphs
CARTE: Pretraining and Transfer for Tabular Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CARTE, a graph-attention transformer that represents each table row as a star graphlet with language-model embeddings of entries and column names, pretrained contrastively on YAGO3, enabling transfer across tables without schema or entity matching.
Problem
Transfer learning and pretraining on tabular data is hard because it requires data integration (schema matching across columns and entity matching across string entries), so tree-based models remain the standard.
Approach
Represent each row as a star-like graphlet whose leaves are cell values and column names embedded by a language model (FastText or e5), and use a graph-attention network with edge-aware attention to contextualize entries; pretrain self-supervised with an InfoNCE contrastive loss on graphlets sampled from the YAGO3 knowledge graph, so unmatched tables can share one representation space.
Key results
- CARTE consistently outperforms alternatives across train sizes on single tables, with the best normalized score and critical-difference ranking against 42 baselines including tree-based models.
- CARTE is more robust to missing features than tree baselines: with 10% dropped features at n=64 its performance drop is 13.28% versus 21.70% for CatBoost.
- On TabLLM datasets, TabPFN performs best, followed by CARTE, XGBoost and TabLLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CARTE instead of the strongest alternatives for single-table learning | CARTE · single-table tasks across regression and classification datasets | normalized score and critical-difference ranking | Helped |
| Add bagging with different train/validation splits for early stopping (ablation) | neural network models · single-table tasks | prediction performance | Helped |
| Add source tables for joint learning | CARTE · related multi-table tasks | target-table performance | Helped |
| Use CARTE rather than CatBoost, TabVec-XGB, or TabVec-RF when features are missing | CARTE · 51 datasets with randomly dropped features | percentage decrease in performance: CARTE 13.28%, 38.35%, 10.19%, 24.42%; CatBoost 21.70%, 53.32%, 12.23%, 29.70% | Helped |
| Use entity string representations without manual entity matching | CARTE · company employees, movies, US accidents, and US election | performance robustness to entity variants | No clear effect |
| Remove attention from the CARTE architecture (ablation) | CARTE · downstream tabular tasks | prediction performance | Hurt |
| Remove string-level representations that capture semantic similarity (ablation) | CARTE · downstream tabular tasks | prediction performance | Hurt |
| Use CARTE instead of CatBoost for single-table learning | CARTE · 51 datasets, learning time at n=64 and n=512 | learning time: 0.98$\pm$1.19 vs 85.43$\pm$60.30; 1.05$\pm$1.06 vs 315.49$\pm$119.84 | Hurt |
Surprising findings
- On the TabLLM-style datasets (mostly numerical, low-cardinality), TabPFN beats CARTE, which the authors attribute to CARTE being geared toward strings.
- CARTE's joint learning across tables without column matching gives consistent gains, whereas CatBoost requires manual column matching to benefit from transfer.
Limitations
- CARTE is much more computationally expensive than tree-based baselines.
- Its advantage is weaker on purely numerical, low-cardinality tables, where TabPFN is stronger.
- Entity-matched results on four datasets were produced by manual matching, which does not scale.
Experimental setup
Datasets: YAGO3 51 tabular datasets from Kaggle, OpenML and prior ML studies TabLLM datasets
Benchmarks: 51 tabular datasets (40 regression, 11 classification) TabLLM nine datasets
Metrics: R2 AUROC normalized score critical difference diagram
Baselines: CatBoost XGBoost HistGradientBoosting RandomForest MLP ResNet TabPFN S-LLM TabLLM
Builds on: TabPFN TabLLM KEN FastText InfoNCE
Closest papers by meaning
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.87)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.87)
- Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.86)
- UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science (ICLR 2024 · similarity 0.85)
- Enhancing Tabular Learners with Context-Aware Semantic Embeddings (NeurIPS 2026 · similarity 0.85)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.