Research map / Vision-language and multimodal / Human and embodied modeling
ConTextTab: A Semantics-Aware Tabular In-Context Learner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConTextTab, a table-native in-context learner that adds semantic embeddings of text cells and column headers (via all-MiniLM-L6-v2) and is pretrained on the real-world T4 dataset, achieving SOTA on the CARTE benchmark.
Problem
Table-native tabular ICL models (TabPFN, TabICL) are trained only on synthetic data and ignore column-name and categorical-value semantics, while LLM-based tabular ICL (TabuLa-8B) captures semantics but can use only ~32 context rows.
Approach
Combine TabPFN-style alternating cross-row/cross-column attention with modality-specific embeddings (pretrained sentence embeddings for text cells and headers, date component sums, clipped numeric embeddings) and pretrain on ~2.18M real-world tables from T4, with optional ISAB blocks for scaling context and a supervised clustering head for classification.
Key results
- ConTextTab achieves the best mean rank on CARTE (1.55) with 76.9% accuracy and 72.4 R^2, statistically significant over all models except CatBoost and AutoGluon.
- On CARTE, ConTextTab outperforms TabPFN (rank 5.76) and TabDPT (rank 5.98) by a wide margin, showing the value of semantic embeddings.
- On non-semantic benchmarks ConTextTab is competitive (e.g., 86.6% on OML-CC18) but behind tuned ensembled boosted trees and RealMLP, which have the best overall ranks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic feature embeddings instead of ordinal encoding (ablation) | ConTextTab · CARTE | accuracy: -2.7 | Helped |
| Add semantic feature embeddings instead of ordinal encoding (ablation) | ConTextTab · CARTE | R²: -4.8 | Helped |
| Add semantic column names rather than dropping column names (ablation) | ConTextTab · CARTE | accuracy: -1.2 | Helped |
| Add LLM-generated descriptions to column names (ablation) | ConTextTab · CARTE | accuracy: 0.0 | No clear effect |
| Increase inference context size (ablation) | ConTextTab · OpenML-CTR23 | R²: +4.2 | Helped |
| Remove inference bagging (ablation) | ConTextTab · CARTE | accuracy: -0.4 | Hurt |
| Use ISAB blocks for cross-row attention (ablation) | ConTextTab · CARTE | R²: -3.1 | Hurt |
| Use non-shared transformer weights (ablation) | ConTextTab · CARTE | R²: -2.6 | Hurt |
| Fine-tune per benchmark with random target selection (ablation) | ConTextTab · OpenML-CTR23 | R²: +7.2 | Helped |
| Increase model size to large (ablation) | ConTextTab · CARTE | accuracy: +0.2 | No clear effect |
| Replace the default sentence embedder with gte-multilingual-base (ablation) | ConTextTab · CARTE | accuracy: +0.0 | No clear effect |
| Replace the cross-entropy head with supervised clustering (ablation) | ConTextTab · CARTE | accuracy: -0.1 | Hurt |
Surprising findings
- Replacing semantic feature encodings with ordinal, MinHash, or Gap encoders drops CARTE accuracy by 1.4 to 4.8 points, while the semantic clustering classification head gave no gain.
- Scaling pretraining data beyond 100k tables and adding a curriculum step on longer-context data brought little gain, and enriching column names with LLM-generated descriptions did not significantly help.
Limitations
- Scaling to very large datasets remains unsolved; all table-native ICL models (ConTextTab, TabPFN, TabDPT, TabICL) fail to match gradient boosting or AutoGluon on large data.
- Evaluation benchmarks lack semantically rich targets and high-cardinality classes, so the clustering head and target semantics could not be properly tested.
- Pretraining on T4 with context limited to 950 rows; inference at 8192 context is extrapolation outside the pretraining setup.
Experimental setup
Datasets: T4 OpenML-CC18 OpenML-CTR23 TALENT TabReD CARTE
Benchmarks: CARTE OpenML-CC18 OpenML-CTR23 TALENT-Tiny TabReD
Metrics: accuracy R^2 (soft-clipped) mean rank win ratio Wilcoxon p-value
Baselines: TabPFN v2.1.0 TabICL v0.1.1 TabDPT CARTE RealMLP TabM XGBoost LightGBM CatBoost AutoGluon Random forest HistGradientBoosting
Builds on: TabPFN TabICL Set Transformer (ISAB) CARTE TabuLa-8B
Closest papers by meaning
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.92)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.91)
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.91)
- Enhancing Tabular Learners with Context-Aware Semantic Embeddings (NeurIPS 2026 · similarity 0.91)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.90)
- Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.