Research map / Architectures and efficiency / Mixture of experts
Mixture of In-Context Prompters for Tabular PFNs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MixturePFN, combining a sparse mixture of K-Means-routed in-context prompters (MICP) for scalable PFN inference with CaPFN, adapter-based finetuning of TabPFN on bootstrapped downstream data, and reports it as the Condorcet winner on 36 TabZilla datasets.
Problem
In-context learning with prior-fitted networks (TabPFN) for tabular data has quadratic time and memory cost in training set size, so it cannot scale beyond about 3,000 samples, and its pretraining prior can mismatch the inference-time data distribution.
Approach
Route each test sample to one of K K-Means-based in-context prompters that each hold a fixed-size (B) support set so test samples can be batched into bounded prompts, and adapt the frozen PFN per dataset with linear adapters trained on bootstrapped sub-datasets drawn from the training data.
Key results
- MixturePFN is the Condorcet winner across 36 TabZilla datasets with 19 wins and 0 losses, with mean log-likelihood rank 2.350 across 20 datasets where all algorithms run.
- On the Top-10 subset (22 datasets), MixturePFN achieves mean log-likelihood rank 2.273 versus 3.955 for CatBoost and 4.045 for TabPFN*, with Wilcoxon significance reported at p<0.05.
- MixturePFN-lite, tuned on only 2 settings, remains Condorcet winner with 503 votes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MICP and CaPFN to TabPFN* | TabPFN* · TabZilla 20 shared datasets | mean rank (log likelihood): 4.550 ± 2.747 -> 2.350 ± 1.824 | Helped |
| add CaPFN bootstrapped adapter finetuning (ablation) | MixturePFN · 10 shared ablation datasets | mean rank: 5.10 ± 1.12 -> 2.75 ± 1.94 | Helped |
| add MICP instead of random sampling (ablation) | TabPFN* with CaPFN · 10 shared ablation datasets | mean rank: 5.30 ± 1.59 -> 2.75 ± 1.94 | Helped |
| replace batched KNN prompts with MICP (ablation) | MixturePFN · 10 shared ablation datasets | mean rank: 7.25 ± 3.70 -> 2.75 ± 1.94 | Helped |
| replace KNN prompting without ensembling with MICP (ablation) | MixturePFN · 10 shared ablation datasets | mean rank: 4.00 ± 1.56 -> 2.75 ± 1.94 | Helped |
| use adapters instead of full finetuning (ablation) | CaPFN · 10 shared ablation datasets | mean rank: 4.85 ± 2.03 -> 2.75 ± 1.94 | Helped |
| increase gamma from 1.0 to 3.0 (ablation) | TabPFN* + MICP · TabZilla datasets | accuracy: 83.96% -> 84.23% | Helped |
| increase gamma from 3.0 to 5.0 (ablation) | TabPFN* + MICP · TabZilla datasets | accuracy: 84.23% -> 84.23% | No clear effect |
Surprising findings
- Adapter-based finetuning on bootstrapped data is reported to outperform full finetuning (MixturePFN CaPFN w. Full FT mean rank 4.85 vs 2.75 for MixturePFN).
Limitations
- Does not scale well with number of features or number of classes; 2 of 36 datasets (more than 10 classes) cannot be run.
- Performance loses to baselines on datasets with many features.
- Evaluated only on classification and the TabZilla benchmark, not huge datasets with billions of samples.
Experimental setup
Datasets: TabZilla benchmark datasets (36 hardest classification datasets)
Benchmarks: TabZilla
Metrics: mean rank Condorcet votes/wins log-likelihood accuracy Wilcoxon signed-rank test
Baselines: TabPFN* XGBoost CatBoost LightGBM SAINT FTTransformer ResNet DANet
Builds on: TabPFN TabZilla Sparse Mixture of Experts KNN prompting
Closest papers by meaning
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.89)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.88)
- TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML 2025 · similarity 0.88)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.88)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.87)
- TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.