Research map / Label-efficient and robust learning / Recognition and long-tail
Making Pre-trained Language Models Great on Tabular Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a RoBERTa-based language model (TP-BERTa) for tabular prediction using relative magnitude tokenization of numerical values and intra-feature attention, achieving performance competitive with GBDTs on typical tabular datasets.
Problem
Transferring pre-trained language models to tabular prediction is hindered by the heterogeneity of tables and by the difficulty LMs have in representing numerical feature values in a discrete text space.
Approach
Discretize numerical values into feature-specific C4.5 bins mapped to shared magnitude tokens scaled by the value, regularize magnitude embeddings with a triplet loss, and fuse each feature name-value pair with a shared intra-feature attention module before the LM so that feature order does not matter.
Key results
- Pre-trained TP-BERTa (Ours_s, default hyperparameters) achieves the best average rank among all methods on binary classification (5.8) and regression (4.3), compared with XGBoost (tuned) at 6.2 and 4.5.
- Substituting RMT with value-string encoding (Value2Str) causes an average AUC drop of 12.45% on significantly changed binary classification datasets; VMFE causes a 3.44% drop.
- Removing the IFA module causes an average AUC decline of 4.17% and a 1.32x training time ratio; pre-training improves over RoBERTa initialization by 2.79% average AUC on significantly changed datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train separately on the corresponding task type rather than jointly on classification and regression | TP-BERTa · Downstream binary classification and regression tasks | average rank: 8.4(4.5) -> 5.8(4.0); 6.9(4.6) -> 4.3(2.8) | Helped |
| Replace RMT with Value2Str numerical encoding (ablation) | TP-BERTa · 80 binary classification datasets | AUC: -12.45% | Hurt |
| Replace RMT with VMFE numerical encoding (ablation) | TP-BERTa · 80 binary classification datasets | AUC: -3.44% | Hurt |
| Remove IFA and feed all feature names and values directly to the LM (ablation) | TP-BERTa · 80 binary classification datasets | AUC: -4.17% | Hurt |
| Reduce magnitude token number from 256 to 32 (ablation) | TP-BERTa · 80 binary classification datasets | AUC: -3.59% | Hurt |
| Reduce magnitude token number from 256 to 128 (ablation) | TP-BERTa · 80 binary classification datasets | AUC: -2.06% | Hurt |
| Add position encoding to value vectors (ablation) | TP-BERTa · 80 binary classification datasets | AUC: -2.35% | Hurt |
| Use tabular pre-training rather than random initialization | TP-BERTa · 80 binary classification datasets | AUC: 3.16% | Helped |
| Use tabular pre-training rather than RoBERTa initialization alone | TP-BERTa · 80 binary classification datasets | AUC: 2.79% | Helped |
Surprising findings
- TP-BERTa performs inferiorly on purely numerical datasets (alpha = 0), where GBDTs or FTT remain better.
- Joint pre-training on classification and regression (Ours_j) performs stably worse than single-task pre-training.
Limitations
- Relies on meaningful feature names; tables with uninformative or encoded names are not well handled.
- Performs worse on purely numerical tables due to precision loss in LM numerical representation.
- Higher parameter count and tuning cost than non-LM tabular DNNs; hyperparameters were fixed to default for fairness.
Experimental setup
Datasets: TabPertNet (101 binary classification and 101 regression pre-training datasets) 80 downstream binary classification datasets 65 downstream regression datasets
Benchmarks: 80 downstream binary classification datasets 65 downstream regression datasets
Metrics: average rank AUC performance change count average AUC difference training time ratio
Baselines: XGBoost CatBoost MLP TabNet AutoInt DCNv2 FT-Transformer SAINT TransTab XTab Value2Str VMFE
Builds on: RoBERTa FT-Transformer TransTab XTab C4.5 discretization
Closest papers by meaning
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.90)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.89)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.88)
- Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning (ICML 2024 · similarity 0.87)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.87)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.