Research map / Architectures and efficiency / RNNs and normalization
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train individual decoders per layer (tabular logit lens) instead of using the original decoder at intermediate layers | 6 tabular foundation models (TabPFN v1/v2/2.5, TabICL, LimiX-2M/16M) · TabArena (15 binary tasks) and PMLBmini (34 tasks) | normalized ROC-AUC | Improved |
| skip an early layer (ablation) | TFMs · binary classification, layer ablation | normalized ROC-AUC | Hurt |
| skip a middle or late layer (ablation) | TFMs · binary classification, layer ablation | normalized ROC-AUC | No clear effect |
| repeat a single layer at inference (ablation) | LimiX-16M and TabPFN(v1) · binary classification, layer ablation | normalized ROC-AUC | Mixed |
| swap adjacent layers (ablation) | TFMs, especially TabPFN(v2) · binary classification, layer ablation | normalized ROC-AUC | Hurt |
| skip the first layer (self-repair check) (ablation) | TFMs · binary classification, tabular logit lens | intermediate ROC-AUC | Hurt |
| skip a middle or late layer (self-repair check) (ablation) | TFMs, especially TabPFN(v2) · binary classification, tabular logit lens | intermediate ROC-AUC | No clear effect |
| use a strong input encoder (TabICL compression, LimiX-2M RBF preprocessing) | TabICL and LimiX-2M · layer skipping | normalized ROC-AUC | Improved |
| use a single transformer layer instead of six (nanoTabPFN) (ablation) | nanoTabPFN · PMLBmini and TabArena | normalized performance | Hurt |
| loop one shared layer up to six times instead of six distinct layers (ablation) | nanoTabPFN (single layer, looped) · PMLBmini and TabArena | normalized performance: uses 20% of the parameters, comparable performance | Improved |
Closest papers by meaning
- Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.90)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.90)
- Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.89)
- Benchmarking Attention for Tabular Foundation Models (NeurIPS 2026 · similarity 0.89)
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.88)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
