Research map / Architectures and efficiency / Pruning and efficient tuning
TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TuneTables, a parameter-efficient prompt-tuning method that compresses large tabular datasets into a small learned context for TabPFN, extending it to larger datasets, more classes, fairness, and interpretability.
Problem
TabPFN and other prior-data fitted networks are limited to about 1000 training samples, 100 features and 10 classes, and their memory and inference cost scales with context length, prohibiting wider adoption on large tabular datasets.
Approach
Apply soft prompt tuning to a pretrained PFN (TabPFN), learning a small set of tuned prompt embeddings that distill a large real dataset into a compact context, optionally with a new decoder for more classes and a demographic parity regularizer for fairness.
Key results
- On the 98-dataset TabZilla suite, TuneTables achieves the highest mean accuracy (0.860) and mean rank (7.435) and the most wins (30), ahead of CatBoost (0.856 acc).
- On LargeScaleTables (datasets up to ~2M samples), TuneTables reaches 0.830 average accuracy versus 0.781 for TabPFNs3000 and 0.823 for CatBoost.
- With a demographic parity objective, TuneTables (Acc+DP) reduces average DP from 0.183 (TabPFN) to 0.109 while keeping average accuracy at 0.840.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize a learned prompt for the downstream dataset | TabPFN · TabZilla (98 datasets) | mean accuracy: 0.856 -> 0.860 | Helped |
| Use medium efficiency modifications (ablation) | TuneTables · LargeScaleTables (29 datasets) | mean accuracy: 0.830 -> 0.828 | Hurt |
| Use medium efficiency modifications (ablation) | TuneTables · LargeScaleTables (29 datasets) | runtime (seconds): 6975 -> 1908 | Helped |
| Use light efficiency modifications and train for one epoch (ablation) | TuneTables · LargeScaleTables (29 datasets) | mean accuracy: 0.828 -> 0.787 | Hurt |
| Use light efficiency modifications and train for one epoch (ablation) | TuneTables-medium · LargeScaleTables (29 datasets) | runtime (seconds): 1908 -> 486 | Helped |
| Tune the prompt for accuracy and demographic parity | TabPFN · Average (Adult, Speeddating, Compas, NLSY) | demographic parity: 0.183 -> 0.109 | Helped |
| Tune the prompt for accuracy and demographic parity | TabPFN · Average (Adult, Speeddating, Compas, NLSY) | accuracy: 0.842 -> 0.840 | Hurt |
| Use smaller tuned prompts (ablation) | TuneTables · Tabular classification datasets, especially smaller datasets | accuracy | Mixed |
| Add ensembling (ablation) | TuneTables · Smaller tabular datasets | accuracy | No clear effect |
| Remove real-data context at inference and use only the tuned prompt (ablation) | TuneTables · Larger tabular datasets | accuracy | Mixed |
| Replace random sketching with k-means or CoreSet sketching (ablation) | TabPFN · Large tabular datasets | accuracy | No clear effect |
| Use PCA for feature reduction instead of random or mutual-information feature selection (ablation) | TabPFN · Datasets with many features | accuracy | Helped |
Surprising findings
- The no-context (NC) setting, where only the tuned prompt is used at test time, is often competitive with or better than using real data as context.
- Random sketching performs as well as k-means and CoreSet for TabPFN subsampling.
Limitations
- Not on par with GBDTs in both accuracy and end-to-end runtime; training time is large.
- Does not improve over TabPFN on small datasets (under 1000 samples, 100 features, 10 classes), postulated to be overfitting.
- Evaluated only on TabPFN as the base PFN.
Experimental setup
Datasets: TabZilla Benchmark Suite (98 datasets) LargeScaleTables (29 datasets) adult-census SpeedDating compas-two-years NLSY Diabetes (100+ class subset)
Benchmarks: TabZilla Benchmark Suite LargeScaleTables Adult SpeedDating COMPAS NLSY
Metrics: accuracy mean rank mean Z-score num. wins demographic parity runtime
Baselines: CatBoost XGBoost LightGBM TabPFNs3000 ExcelFormer SAINT FT-Transformer ResNet Random Forest SVM
Builds on: TabPFN soft prompt tuning last-layer retraining Optuna
Closest papers by meaning
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.89)
- TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.88)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.88)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.88)
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.88)
- A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.