Research map / Probabilistic and scientific modeling / Causal discovery
Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Drift-Resilient TabPFN pre-trains a TabPFN-style prior-data fitted network on synthetic SCM datasets whose edge weights shift over time (via a secondary SCM), enabling in-context prediction under temporal distribution shift on small tabular data.
Problem
Tabular classifiers assume i.i.d. data and degrade under temporal distribution shifts; existing tabular and temporal domain generalization methods (GBDTs, Wild-Time NN methods) fail to consistently generalize out-of-distribution, especially on small datasets.
Approach
Extend the TabPFN SCM prior so that sparse subsets of causal edge weights drift over time, with the drift generated by a secondary random SCM conditioned on a Time2Vec-encoded domain index; the transformer then learns in-context to extrapolate decision boundaries to future domains in a single forward pass.
Key results
- On the combined datasets, the method reaches OOD accuracy 0.744 and ROC AUC 0.832 (vs. 0.688 and 0.786 for TabPFN-base) as reported in the abstract's comparison.
- On synthetic OOD data, accuracy rises from 0.665 (CatBoost) to 0.754 and ECE drops from 0.164 to 0.126 relative to the strongest baseline.
- On real-world OOD data, accuracy improves from 0.712 to 0.736 and ROC AUC from 0.82 to 0.822; the method takes on average 10.9s for training and prediction combined.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the temporal-shift SCM prior instead of the standard TabPFN prior | TabPFN · Synthetic + real-world datasets, OOD | accuracy: 0.688 -> 0.744 | Helped |
| Use the temporal-shift SCM prior instead of the standard TabPFN prior | TabPFN · Synthetic + real-world datasets, OOD | ROC AUC: 0.786 -> 0.832 | Helped |
| Use the temporal-shift SCM prior instead of the standard TabPFN prior | TabPFN · Synthetic + real-world datasets, OOD | ECE: 0.119 -> 0.091 | Helped |
| Add Time2Vec preprocessing (ablation) | Drift-Resilient TabPFN · Synthetic + real-world datasets, OOD | accuracy: 0.742 -> 0.744 | No clear effect |
| Add Time2Vec preprocessing (ablation) | Drift-Resilient TabPFN · Synthetic + real-world datasets, OOD | ROC AUC: 0.832 -> 0.832 | No clear effect |
| Shift domain boundaries probabilistically (ablation) | Drift-Resilient TabPFN · Validation datasets, OOD | accuracy: 0.74 -> 0.734 | Hurt |
| Shift boundaries and probabilistically merge domains (ablation) | Drift-Resilient TabPFN · Validation datasets, OOD | accuracy: 0.74 -> 0.742 | No clear effect |
| Shift boundaries, merge domains, and add noise to domain indicators (ablation) | Drift-Resilient TabPFN · Validation datasets, OOD | accuracy: 0.74 -> 0.704 | Hurt |
Surprising findings
- Ablation shows the gains come mainly from the shift-aware prior rather than Time2Vec preprocessing, which adds only a statistically insignificant improvement.
- Perturbing temporal domain indices during pre-training generally degrades performance, indicating the model needs accurate domain indices.
Limitations
- Quadratic attention scaling limits the method to small datasets.
- Black-box transformer makes predictions and recognized shifts hard to interpret.
- The SCM prior with sparse mechanism shifts may not describe all real-world shifts.
Experimental setup
Datasets: 18 test datasets (8 synthetic, 10 real-world) 12 validation datasets (4 synthetic, 8 real-world) synthetic SCM prior datasets
Benchmarks: 18 synthetic and real-world tabular datasets (Eval-Fix protocol from Wild-Time) Rotated Two Moons (vs DRAIN, GI)
Metrics: accuracy F1 ROC AUC ECE
Baselines: TabPFN base CatBoost XGBoost LightGBM Wild-Time ERM Wild-Time SWA DRAIN GI
Builds on: TabPFN Prior-Data Fitted Networks Time2Vec Wild-Time
Closest papers by meaning
- Do-PFN: In-Context Learning for Causal Effect Estimation (NeurIPS 2025 · similarity 0.87)
- A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · similarity 0.87)
- TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.86)
- TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments (ICML 2025 · similarity 0.86)
- Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.86)
- Learning Causal Orderings for In-Context Tabular Prediction (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.