Research map / Label-efficient and robust learning / Recognition and long-tail

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TabPrep, a lightweight pattern-informed feature generation pipeline (OAFE, CatPrep, RSFC) that, added to tabular models, sets new peak performance on TabArena and shows a "feature engineering gap" in model-only benchmarks.

Problem

Modern tabular benchmarks omit feature engineering, so model-only comparisons underestimate attainable peak performance; existing automated feature engineering libraries are too expensive and fail on many datasets for large-scale benchmarking.

Approach

Replace exhaustive expand-and-reduce autoFE with budgeted generation of features targeting three recurring structural patterns (arithmetic interactions, group-conditional categorical effects, pseudo-categorical numeric interactions), using OOF target encoding and random subset compression, treated as a hyperparameter in model tuning.

Key results

• TabPrep-augmented models set new dataset-level peak performance on 24 of 51 TabArena datasets (19 relative to TabPFN-2.6 and TabICLv2 alone).
• PrepTabPFN-2.5 outperforms all other TabArena competitors, including TabPFN-2.6 and TabICLv2, in average Elo.
• PrepLightGBM wins 85% of pairwise comparisons against tuned LightGBM, and PrepLinear 97% against the linear model; median training-time overhead is 6x for LightGBM versus 180x for autofeat and 19x for OpenFE.

Surprising findings

• autofeat and OpenFE fail on 33 and 22 datasets respectively due to memory or time limits, while TabPrep runs on all datasets.
• TabPrep benefits are even larger on medium-sized datasets (10,000-100,000 training samples), enabling LightGBM to outperform AutoGluon's multi-model ensemble there.

Limitations

• TabPrep is a benchmark baseline and does not find optimal features, so gains are a lower bound on feature engineering potential.
• Only four model families are evaluated, and evaluation is restricted to IID tabular tasks without text or relational data.
• Feature expansion increases training and inference cost, and for TabM and TabPFN-2.5 always-on TabPrep is not always better than baseline.

Experimental setup

Datasets: TabArena (51 datasets)

Benchmarks: TabArena

Metrics: Elo improvability normalized score relative improvement harmonic mean rank win rate

Baselines: autofeat OpenFE TabPFN-2.6 TabICLv2 LightGBM TabM TabPFN-2.5 AutoGluon linear model

Builds on: TabArena OpenFE autofeat CatBoost TabPFN-2.5

Closest papers by meaning

• A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.91)
• TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments (ICML 2025 · similarity 0.91)
• TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.90)
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning (NeurIPS 2024 · similarity 0.89)
• TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.88)
• Tabular Foundation Model for Generative Modelling (NeurIPS 2026 · similarity 0.88)