Research map / Large language models / Language, safety and interpretability
Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FeatLLM, which uses an LLM (GPT-3.5) in-context to generate rule-based binary features per class, then trains a non-negative linear model with bagging ensembles, enabling few-shot tabular prediction without per-sample LLM queries.
Problem
Few-shot tabular learning with LLMs is expensive because end-to-end methods need an LLM call per sample, require fine-tuning or training access, and struggle with long prompts when many features are present.
Approach
Use the LLM as a feature engineer: a structured prompt (task description, reasoning in two steps, response format) elicits per-class rules from few-shot examples; the LLM itself converts rules into code producing binary features; a non-negative linear model over these features is trained, and bagged ensembles of such runs are averaged, so inference needs no LLM calls.
Key results
- Averaged AUC over 13 datasets at 4 shots: FeatLLM 77.86 vs. TabLLM 70.26 and In-context 68.44 (per the main 11-dataset table averages).
- On Communities and Myocardial (over 100 attributes), FeatLLM is the top method at 4 shots on Communities (75.39 AUC) where other LLM-based baselines failed to run.
- Inference over one sample takes 0.006 ms for FeatLLM versus 463 ms for in-context learning, with 30 API queries used for training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FeatLLM instead of TabLLM | Few-shot tabular learning · 13 datasets, 4 shots | AUC: 70.26 -> 77.86 | Helped |
| Use FeatLLM instead of TabLLM | Few-shot tabular learning · 13 datasets, 8 shots | AUC: 72.76 -> 79.31 | Helped |
| Use FeatLLM instead of TabLLM | Few-shot tabular learning · 13 datasets, 16 shots | AUC: 76.22 -> 80.70 | Helped |
| Add weight tuning to the linear model (ablation) | FeatLLM · 13 datasets, average across shots | AUC: -3.47$\pm$0.51 | Helped |
| Add ensemble (ablation) | FeatLLM · 13 datasets, average across shots | AUC: -6.49$\pm$0.51 | Helped |
| Add feature descriptions (ablation) | FeatLLM · 13 datasets, average across shots | AUC: -0.84$\pm$0.28 | Helped |
| Add Step-1 reasoning when generating rules (ablation) | FeatLLM · 13 datasets, average across shots | AUC: -2.84$\pm$0.53 | Helped |
| Add bagging (ablation) | FeatLLM rule generation · Rule sets across iterations | Correlation: 0.384 $\pm$ 0.327 -> 0.368 $\pm$ 0.341 | Helped |
| Add instance shuffling (ablation) | FeatLLM rule generation · Rule sets across iterations | Correlation: 0.531 $\pm$ 0.290 -> 0.368 $\pm$ 0.341 | Helped |
| Use nonzero temperature instead of zero temperature (ablation) | FeatLLM rule generation · Rule sets across iterations | Correlation: 0.414 $\pm$ 0.373 -> 0.368 $\pm$ 0.341 | Helped |
| Increase the number of ensembles | FeatLLM · Tabular datasets | AUC | Helped |
Surprising findings
- Flipping training labels still leaves the rule set largely driven by prior knowledge, with high variance across trials.
- Error rate of LLM-generated parsing code reaches up to 40% on datasets with many categorical features (e.g., Credit-g, Bank).
Limitations
- Only designed for low-shot regimes; not evaluated on larger sample sizes.
- Relies on a proprietary LLM API and reported parse failures up to 40% for some datasets.
- Rules are restricted to binary feature form, limiting interpretability and feature types.
Experimental setup
Datasets: Adult Bank Blood Car Communities Credit-g Diabetes Heart Myocardial Cultivars NHANES Sequence-type
Benchmarks: Adult Bank Blood Car Communities Credit-g Diabetes Heart Myocardial Cultivars NHANES Sequence-type
Metrics: AUC
Baselines: LogReg XGBoost RandomForest SCARF STUNT TabPFN In-context learning TABLET TabLLM
Builds on: chain-of-thought TabLLM TABLET STUNT TabPFN
Closest papers by meaning
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.89)
- Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning (NeurIPS 2024 · similarity 0.89)
- TabFlex: Scaling Tabular Learning to Millions with Linear Attention (ICML 2025 · similarity 0.88)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.88)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.88)
- Making Pre-trained Language Models Great on Tabular Prediction (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.