Research map / Architectures and efficiency / RNNs and normalization
EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EquiTabPFN, a tabular prior-fitted network that is equivariant to target-dimension permutations, enabling in-context classification with arbitrary numbers of classes without costly ensembling.
Problem
TabPFN-style tabular foundation models are limited to a fixed pre-defined number of target dimensions (classes) and are not target-permutation equivariant, so predictions depend on class ordering and handling more classes requires costly ensembling.
Approach
Designs a fully target-equivariant PFN architecture: per-class target tokens embedded by a shared 1x1 convolution, alternating attention across components and across datapoints (bi-attention), prediction tokens for test targets, and a non-parametric attention-based decoder with residual pointwise MLP. The paper also defines a target-equivariance gap and shows minimizers of the pre-training objecti
Key results
- On 10 datasets with more than 10 classes, EquiTabPFN obtains the best median relative accuracy, strongly outperforming TabPFNv2 (ECOC ensembling), which performs worse than linear models or random forests in median relative terms.
- On 76 datasets with fewer than 10 classes, EquiTabPFN performs comparably to TabPFNv2 (median relative accuracy 6.3 vs 9.1 for TabPFNv2 in the table) and outperforms the retrained TabPFNv2* (4.3).
- EquiTabPFN runtime on >10-class datasets is 0.40 s vs 2.80 s for TabPFNv2, while using ~25M parameters, and it uses about 7.45x more FLOPS than TabPFN on 10-class data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use target-equivariant EquiTabPFN instead of TabPFNv2 | TabPFN-style classification · TabZilla, datasets with >10 classes | Median relative Acc.: 9.1 -> 6.3 | Hurt |
| use target-equivariant EquiTabPFN instead of TabPFNv2 | TabPFN-style classification · TabZilla, datasets with >10 classes | Mean time (s): 0.2 -> 0.1 | Helped |
| use target-equivariant EquiTabPFN instead of TabPFNv2 | TabPFN-style classification · TabZilla, datasets with <10 classes | Mean Acc.: 74.5 +/- 2.6 -> 77.0 +/- 1.8 | Helped |
| use target-equivariant EquiTabPFN instead of TabPFNv2 | TabPFN-style classification · TabZilla, datasets with <10 classes | Median relative Acc.: 6.0 -> 27.9 | Helped |
| use target-equivariant EquiTabPFN instead of TabPFNv2 | TabPFN-style classification · TabZilla, datasets with <10 classes | Mean time (s): 2.8 -> 0.4 | Hurt |
| add the equivariant decoder to the TabPFN backbone (ablation) | TabPFN backbone · TabZilla benchmark tasks (ablation evaluation) | % Error reduction over TabPFN: +0.94% | Helped |
| add the bi-attention backbone to the MLP-decoder configuration (ablation) | TabPFNv1 MLP decoder · TabZilla benchmark tasks (ablation evaluation) | % Error reduction over TabPFN: -0.12% | No clear effect |
| combine bi-attention with the equivariant decoder (ablation) | EquiTabPFN · TabZilla benchmark tasks (ablation evaluation) | % Error reduction over TabPFN: +1.50% | Helped |
| add residual correction and target-dependent embeddings to the decoder (ablation) | EquiTabPFN decoder · classification benchmarks | performance | Helped |
Surprising findings
- TabPFNv2 with ECOC ensembling performs worse than a linear model or random forests on datasets with more than 10 classes.
- Even trained with an equivariance-free objective, TabPFN's equivariance error decreases only slowly during training, and ensembling over permutations needs many members to approach equivariance.
Limitations
- Quadratic extra cost in the number of target dimensions due to self-attention across components.
- Assumes targets are permutation-invariant; ordinal targets (5 of 86 surveyed datasets) violate this assumption.
- Pre-trained only on classification tasks with fewer than 10 classes and a public prior, and TabPFNv2 comparison uses a retrained variant because the official prior/training code is unavailable.
Experimental setup
Datasets: synthetic datasets from TabPFNv1 prior TabZilla benchmark classification datasets (76 with <10 classes, 10 with >10 classes)
Benchmarks: TabZilla 76 classification datasets (<10 classes) TabZilla 10 classification datasets (>10 classes)
Metrics: median relative accuracy over KNN accuracy AUC F1 runtime
Baselines: TabPFNv1 TabPFNv2 TabPFNv2* XGBoost rtdl(ResNet) Random Forest KNN
Builds on: TabPFN TabPFNv2 bi-attention for feature equivariance
Closest papers by meaning
- A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities (NeurIPS 2025 · similarity 0.87)
- TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems (ICML 2025 · similarity 0.86)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.86)
- Learning (Approximately) Equivariant Networks via Constrained Optimization (NeurIPS 2025 · similarity 0.86)
- TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks (NeurIPS 2024 · similarity 0.85)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.