Research map / Large language models / Language, safety and interpretability
TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TabFSBench, the first benchmark for feature shifts (feature decrement) in tabular data, evaluating 4 model categories including LLMs and tabular LLMs under single, most/least-relevant, and random shift scenarios.
Problem
Tabular models are typically evaluated in closed environments with fixed feature sets; feature shifts (features added or removed between training and testing) in open environments degrade performance and have received little study or benchmarking.
Approach
Simulates feature-decrement shifts by removing features (ranked by Pearson correlation with the target, or randomly) at test time and imputing them with training-set mean/mode, then measures relative performance gap across tree-based, deep-learning, LLM and tabular-LLM models.
Key results
- Tree-based models degrade most under feature shift; on binary classification at 100% shift LightGBM drops by -0.353 relative gap while the best (UniPredict) drops -0.080.
- Tabular LLM UniPredict and TabLLM achieve the best average ranks (2.9 and 3.3) across classification and regression scenarios, while CatBoost ranks best among closed-environment performers in several tasks.
- Shifted feature set correlation sum has a linear relationship with performance degradation (Pearson rho = 0.7405).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove features at test time, with shift degree increasing | Tabular models · Binary and multi-class classification; regression | performance | Hurt |
| Remove a single feature with stronger correlation to the target | Tabular models · 12 datasets | performance | Hurt |
| Remove features with low target relevance | Tabular models · 12 datasets | performance | Mixed |
| Shift a feature set with a larger sum of feature correlations | Tabular models · Feature-shift scenarios | performance degradation: ρ = 0.74 | Hurt |
| Use a model with stronger closed-environment performance | Tabular models · Random feature shifts | performance | Helped |
| Use TabLLM or UniPredict rather than most other models | Tabular models · Classification feature shifts | performance gap Δ | Helped |
| Use DANets or TabTransformer rather than most other models | Tabular models · Regression feature shifts | performance gap Δ | Helped |
| Use mean imputation instead of random imputation | Tabular models · Feature-decrement scenarios | performance | Helped |
| Use tabular LLMs for robust feature-shift performance | Tabular models · Across feature-shift scenarios | performance | Mixed |
Surprising findings
- Tabular LLMs show the smallest performance degradation under feature shift despite not always being best in closed environments.
- Removing least-relevant features can sometimes improve model performance.
Limitations
- Focuses only on feature-decrement scenarios; feature-increment and specialized feature-incremental architectures are not evaluated.
- Imputation with training mean/mode is a simplifying assumption and may not reflect real missingness; the analysis does not cover shifted features with identical correlation structures.
- Only one LLM (Llama3-8B) and two tabular LLMs are evaluated, limiting conclusions about LLMs.
Experimental setup
Datasets: credit electricity heart MiniBooNE Iris penguins eye_movements jannis abalone bike concrete laptop
Benchmarks: TabFSBench
Metrics: accuracy ROC-AUC RMSE relative performance gap (Delta) average rank
Baselines: LightGBM XGBoost CatBoost FT-Transformer TabPFN TabLLM UniPredict Llama3-8B
Builds on: TabZilla LAMDA-Talent TabLLM UniPredict
Closest papers by meaning
- TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks (NeurIPS 2026 · similarity 0.91)
- TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.90)
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.90)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.89)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.89)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.