Research map / Large language models / Instruction and fine-tuning
Large Scale Transfer Learning for Tabular Data via Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Llama 3-8B on a filtered 100B-token tabular corpus (T4 from TabLib) with a row-causal tabular attention mask to get TabuLa-8B, a training-free in-context tabular predictor that beats XGBoost and TabPFN in few-shot settings.
Problem
Tabular prediction usually requires task-specific models trained on the target data; transfer learning has not matched the impact seen in language and vision, and LLM-based tabular predictors have lacked large-scale training data and out-of-distribution competitiveness.
Approach
Build a large, filtered corpus of tables (T4 from TabLib), serialize rows as text with unsupervised target selection, and fine-tune Llama 3-8B with a row-causal tabular masking (RCTM) scheme that lets rows of the same table attend to each other within a packed batch, so the model learns to use multiple shots in context.
Key results
- TabuLa-8B reaches zero-shot accuracy over 15 pp above random guessing on 329 evaluation datasets, which the paper says XGBoost and TabPFN cannot do.
- In the 1-32 shot regime, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN trained on equal or more data.
- TabuLa-8B reaches 60% average accuracy on UniPredict with one shot, versus 16 shots for XGBoost and TabPFN (a 16x relative sample efficiency).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama 3-8B on T4 for tabular prediction | Llama 3-8B · Five tabular benchmarks, zero-shot | accuracy relative to random guessing: 17 percentage points above random guessing | Helped |
| Use TabuLa-8B instead of XGBoost or TabPFN | TabuLa-8B · Five tabular benchmarks, 1-32 shots | accuracy: 5-20pp | Helped |
| Use TabuLa-8B instead of Llama 3-8B | Llama 3-8B · Five tabular benchmarks, few-shot | accuracy: 10 to 20pp | Helped |
| Add row-causal tabular masking (RCTM) instead of per-sample causal attention (ablation) | TabuLa-8B · Full test suite, few-shot | accuracy | Helped |
| Replace informative column headers with X1, X2, ... and target header Y (ablation) | TabuLa-8B · UniPredict, 16 shots | accuracy: 3-5pp | Hurt |
| Replace informative column headers with uninformative headers (ablation) | TabuLa-8B · UniPredict, 32 shots | accuracy | No clear effect |
| Remove features at evaluation, especially unimportant features (ablation) | TabuLa-8B · Random sample of 32 datasets, 0-, 4-, 16-, and 32-shot | accuracy | Mixed |
| Randomly permute columns (ablation) | TabuLa-8B · Random subset of 32 tasks, 0, 8, 16, and 32 shots | accuracy | No clear effect |
| Use full data filtering instead of minimal filtering (ablation) | TabuLa-8B · Tabular test suite, 16 or more shots | accuracy | Helped |
| Use full data filtering instead of minimal filtering (ablation) | TabuLa-8B · Tabular test suite, fewer than 16 shots | accuracy | No clear effect |
| Use a stronger base language model, from Llama 1 through Llama 2 to Llama 3 (ablation) | TabuLa-8B training pipeline · Tabular test suite, up to 8 shots | accuracy | Helped |
| Evaluate separately on potentially contaminated versus uncontaminated tables | TabuLa-8B · Evaluation test suite | accuracy | No clear effect |
Surprising findings
- Claude Instant and Claude 3 Sonnet show little to no improvement with more shots and perform worse than most baselines beyond 3 shots.
- Removing informative column headers causes only a small drop, and the header benefit vanishes at 32 shots.
Limitations
- Context window limited to 8192 tokens, which restricts the number of shots and extra information.
- 8B parameters make serving and inference expensive.
- Possible social biases from web-scale historical data introduce fairness concerns; up to one-third of benchmark tables may appear in the training corpus.
Experimental setup
Datasets: T4 (filtered from TabLib) UniPredict Grinsztajn benchmark AutoML Multimodal Benchmark OpenML CC-18 OpenML CTR-23
Benchmarks: UniPredict Grinsztajn AutoML Multimodal Benchmark OpenML CC-18 OpenML CTR-23
Metrics: open-vocabulary accuracy zero-shot accuracy few-shot accuracy relative sample efficiency
Baselines: Llama 3-8B XGBoost TabPFN CatBoost Logistic Regression Claude Instant Claude 3 Sonnet
Builds on: Llama 3 TabPFN XGBoost TabLib in-context pretraining
Closest papers by meaning
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.91)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.91)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.90)
- Making Pre-trained Language Models Great on Tabular Prediction (ICLR 2024 · similarity 0.90)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.90)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.