Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APT, a TabPFN-style prior-data fitted network pre-trained on synthetic data from a mix of random MLP generators and adversarial data agents, plus a scatter-sum mixture block that removes the fixed class-count limit for zero-shot tabular classification and regression.
Problem
Zero-shot meta-learning on tabular prediction is limited by narrow synthetic pre-training distributions in PFNs like TabPFN and by a fixed maximum number of classes in the output head.
Approach
Adversarial synthetic data agents (generators updated by gradient ascent on the meta-learner's loss, reset periodically) diversify PFN pre-training data, and a scatter-sum attention-based mixture block replaces the dense final layer so any number of classes can be handled.
Key results
- APT achieves mean ROC-AUC 0.921 and mean rank 3.86 on 35 small OpenML-CC18 datasets, the best among compared methods (TabPFN 0.913, CatBoost 0.918), with mean runtime 0.90 s versus about 0.86 s for TabPFN.
- On OpenML-CTR23 regression ablation, APT (adversarial pre-training on TabPFN) improves mean MSE from 0.340 to 0.306 on all datasets and wins 28.6 vs 6.4 for TabPFN.
- With the mixture block, APT reaches AUC 0.90 in 11,780 gradient steps; the mixture block accelerates pre-training substantially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adversarial pre-training instead of ordinary TabPFN pre-training (ablation) | TabPFN · OpenML-CC18 small datasets | mean AUC: 0.913 ± 0.003 -> 0.921 ± 0.003 | Helped |
| add mixture block (ablation) | APT · OpenML-CC18 small datasets | mean AUC: 0.917 ± 0.005 -> 0.921 ± 0.003 | Mixed |
| add mixture block without adversarial pre-training (ablation) | TabPFN · OpenML-CC18 small datasets | mean AUC: 0.913 ± 0.003 -> 0.914 ± 0.004 | No clear effect |
| add adversarial pre-training (ablation) | TabPFN · OpenML-CTR23 small datasets | mean MSE: 0.412 ± 0.077 -> 0.344 ± 0.068 | Helped |
| add adversarial pre-training (ablation) | TabPFN · OpenML-CTR23 all datasets | mean MSE: 0.340 ± 0.025 -> 0.306 ± 0.023 | Helped |
| use mixture block instead of a dense classification head (ablation) | APT · OpenML-CC18 datasets with more than 10 classes | ROC-AUC: letter: .975 ± .002 -> .975 ± .002; isolet: .970 ± .003 -> .939 ± .011; vowel: 1 ± 0 -> 1 ± 0; texture: 1 ± 0 -> 1 ± 0 | Mixed |
| use adversarial data agents rather than ordinary random generators | APT pre-training · synthetic two-dimensional datasets | KL divergence between generated dataset collections: 0.134 ± 0.141 -> 0.813 ± 0.072 | Helped |
Surprising findings
- Adversarial data agents produced synthetic data with slightly stronger predictor-response correlation (0.311) than ordinary generators (0.268), contrary to the expectation that adversarial training would weaken signal.
- The APT vs. CatBoost comparison on categorical-heavy datasets shows GBDTs still lead, and APT underperforms GBDTs on large datasets.
Experimental setup
Datasets: OpenML-CC18 OpenML-CTR23 synthetic data from sparsified noisy random MLPs and adversarial agents
Benchmarks: OpenML-CC18 (35 small datasets) OpenML-CTR23 (12 small / 35 total regression datasets)
Metrics: ROC-AUC (OVO) rank wins runtime MSE KL divergence
Baselines: TabPFN CatBoost XGBoost LightGBM SVM SAINT Tabular ResNet Random Forest LASSO-Logistic Ridge-Logistic KNN
Builds on: TabPFN Prior-Data Fitted Networks GANs Gumbel-Softmax Concrete distribution
Closest papers by meaning
- RAD-TFM: Robust and Domain-Adapted Tabular Foundation Models (NeurIPS 2026 · similarity 0.86)
- CG-TGAN: Conditional Generative Adversarial Networks with Graph Neural Networks for Tabular Data Synthesizing (AAAI 2025 · similarity 0.85)
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.85)
- A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · similarity 0.84)
- Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.84)
- TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.