Research map / Large language models / Language, safety and interpretability
The Illusion of Generalization in Tabular Language Models
Aditya Gorla, Ratish Puduppully
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add general instruction tuning (Alpaca) to Base Llama (ablation) | Llama-3-8B · UniPredict classification (119 datasets) | accuracy (%): 47.7 -> 58.6 | Improved |
| add tabular pretraining on T4 on top of instruction tuning (Tabula-8B vs Alpaca) (ablation) | Alpaca · UniPredict classification (119 datasets) | accuracy (%): 58.6 -> 63.5 | Improved |
| exclude 5 contaminated datasets from the decomposition (ablation) | instruction-tuning contribution share · UniPredict classification | share of gain (%): 69% -> 73% | Mixed |
| add quartile-format examples to Alpaca (Alpaca+Q) (ablation) | Alpaca · UniPredict quartile classification | lift (pp): 5.6 -> 25.8 | Improved |
| add tabular pretraining (Tabula-8B vs Alpaca+Q) on non-stock quartile datasets (ablation) | Alpaca+Q · UniPredict quartile non-stock | lift (pp): 6.7 -> 6.5 | No clear effect |
| add tabular pretraining (Tabula-8B vs Alpaca+Q) on stock/financial quartile datasets (ablation) | Alpaca+Q · UniPredict quartile stock/financial | lift (pp): 48.2 -> 66.1 | Improved |
| report lift over majority baseline instead of raw accuracy | Tabula-8B · Stroke Prediction | accuracy vs baseline: 0.949 accuracy vs 0.949 majority | Hurt |
| evaluate on binary instead of quartile tasks | Tabula-8B · UniPredict | median lift (pp): 32.9 -> 2.5 | Hurt |
| evaluate on categorical instead of quartile tasks | Tabula-8B · UniPredict | median lift (pp): 32.9 -> -0.3 | Hurt |
| use Base Llama without instruction tuning | Llama-3-8B · UniPredict classification | datasets at or below majority baseline (%): 60.5 -> 75.6 | Hurt |
Closest papers by meaning
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.93)
- Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.90)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.90)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.89)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.89)
- Making Pre-trained Language Models Great on Tabular Prediction (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
