Research map / Large language models / Instruction and fine-tuning
Retrieval & Fine-Tuning for In-Context Tabular Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoCalPFN, which adds kNN retrieval and per-dataset end-to-end fine-tuning on top of TabPFN so each query is classified with a local context, achieving state-of-the-art AUC on 95 TabZilla/OpenML datasets.
Problem
In-context learning tabular models like TabPFN struggle to scale to large datasets (quadratic context memory, subsampling) and to underfit complex decision boundaries when using a single global context.
Approach
Replace the global random context of TabPFN with each query's k nearest neighbours from the training set (TabPFN-kNN), and fine-tune the model end-to-end with approximate local contexts that share neighbour sets across batches of queries, yielding LoCalPFN.
Key results
- LoCalPFN reaches 0.958 IQM AUC and 0.908 mean AUC over all 95 datasets, versus 0.917 IQM AUC for vanilla TabPFN and 0.943 for XGBoost.
- On 48 medium/large datasets, LoCalPFN reaches 0.968 IQM AUC versus 0.927 for TabPFN and 0.966 for LightGBM.
- Fine-tuning without retrieval plus kNN does not improve further, while joint retrieval and fine-tuning (LoCalPFN) gives the best results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add retrieval and task-specific fine-tuning to TabPFN | TabPFN · TabZilla 95 datasets (All) | IQM AUC: 0.917 -> 0.958 | Helped |
| retrieve nearest neighbours as the context | TabPFN · TabZilla medium/large datasets | IQM AUC: 0.927 -> 0.955 | Helped |
| jointly fine-tune with retrieval rather than fine-tuning on random context and applying kNN afterward (ablation) | TabPFN · TabZilla 95 datasets (All) | IQM AUC: 0.938 -> 0.958 | Helped |
| add fine-tuning without retrieval (ablation) | TabPFN · TabZilla 95 datasets (All) | IQM AUC: 0.917 -> 0.934 | Helped |
| apply kNN retrieval after random-context fine-tuning (ablation) | TabPFN Fine-Tune · TabZilla 95 datasets (All) | IQM AUC: 0.934 -> 0.938 | Hurt |
| use one-hot instead of raw feature encoding for retrieval (ablation) | LoCalPFN · TabZilla 95 datasets (All) | IQM AUC: 0.956 -> 0.958 | Helped |
| use TabPFN encoder embeddings instead of one-hot encoding for retrieval (ablation) | LoCalPFN · TabZilla 95 datasets (All) | IQM AUC: 0.958 -> 0.955 | Hurt |
| use an ensemble of 32 models with randomized feature and class ordering (ablation) | TabPFN · TabZilla 95 datasets (All) | IQM AUC: 0.917 -> 0.936 | Helped |
| increase ensembled TabPFN context size to 3,000 (ablation) | TabPFN-1k-32ens · TabZilla 95 datasets (All) | IQM AUC: 0.936 -> 0.943 | Helped |
| average predictions over training-data context chunks (ablation) | TabPFN-3k-32ens · TabZilla 95 datasets (All) | IQM AUC: 0.943 -> 0.945 | Helped |
| use a maximum neighbour count of 50 instead of 20 (ablation) | TabPFN-kNN · TabZilla medium/large datasets | IQM AUC: 0.937 -> 0.949 | Helped |
| increase LoCalPFN maximum neighbour count from 50 to 1000 (ablation) | LoCalPFN · TabZilla medium/large datasets | IQM AUC: 0.960 -> 0.968 | Helped |
Surprising findings
- Applying TabPFN-kNN on top of a naively fine-tuned TabPFN does not improve performance further, indicating joint retrieval and fine-tuning is needed.
- LoCalPFN with a maximum of only 50 neighbours outperforms the random forest baseline on the medium/large datasets.
Limitations
- Only validated with TabPFN as base model; transfer to other in-context tabular learners is untested.
- Limited to classification tasks with TabPFN's feature and class constraints (at most 100 features, 10 classes); regression not addressed.
- LoCalPFN has higher runtime than tree-based methods.
Experimental setup
Datasets: TabZilla 95 OpenML datasets
Benchmarks: TabZilla 95 OpenML datasets 71-dataset subset for deep learning comparison
Metrics: AUC (IQM and mean) accuracy F1
Baselines: TabPFN TabPFN 3k XGBoost CatBoost LightGBM RandomForest kNN TabNet rtdl_FTTransformer TabPFN-ICD
Builds on: TabPFN RAG TabR prior-fitting
Closest papers by meaning
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.92)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.91)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.90)
- Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.90)
- TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.90)
- A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.