Research map / Large language models / Agents, code and math
Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OCTree, which uses LLMs as optimizers to iteratively generate column-feature rules for tabular data, feeding back validation scores and CART decision-tree reasoning from past experiments.
Problem
Automated feature engineering for tabular prediction typically relies on pre-defined search spaces and validation-score-only feedback, discarding insights from past experiments.
Approach
Use an LLM as a black-box optimizer over feature-generation rules, with a trajectory of past rules, validation scores, and natural-language CART decision-tree reasoning fitted on the augmented training data as feedback; no manually specified search space is needed.
Key results
- On the 19 datasets without language descriptions, OCTree reduces XGBoost average error from 18.30% to 17.45% (4.6% relative), and to 16.85% (7.9%) when combined with OpenFE.
- With GPT-4o on datasets with language descriptions, OCTree reduces XGBoost error on Tesla Stock by 17.1% relative (6.61 to 5.48 MAE x1e-3), outperforming CAAFE (3.8%).
- On the madelon dataset (501 features), OCTree reduces XGBoost error by 6.3% relative (21.54% to 20.19%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate features with OCTree instead of baseline XGBoost feature engineering | XGBoost · 22-dataset average | error rate (%): 4.6% | Helped |
| Apply OpenFE after OCTree feature generation | XGBoost · 22-dataset average | error rate (%): 7.9% | Helped |
| Generate features with OCTree instead of baseline MLP feature engineering | MLP · 22-dataset average | error rate (%): 4.6% | Helped |
| Apply OpenFE after OCTree feature generation | MLP · 22-dataset average | error rate (%): 7.0% | Helped |
| Generate features with OCTree using GPT-4o rather than the baseline | XGBoost · Tesla | MAE (×10^-3): 17.1% | Helped |
| Add decision tree reasoning feedback (ablation) | OCTree with XGBoost · Disease | test error rate (%): 27.62 -> 26.19 | Helped |
| Add decision tree reasoning feedback (ablation) | OCTree with XGBoost · Clinical | test error rate (%): 45.61 -> 45.07 | Helped |
| Add decision tree reasoning feedback (ablation) | OCTree with XGBoost · electricity | test error rate (%): 6.89 -> 6.65 | Helped |
| Add decision tree reasoning feedback (ablation) | OCTree with XGBoost · kddCup09 | test error rate (%): 19.47 -> 19.07 | Helped |
| Transfer OCTree-generated features from XGBoost to MLP (ablation) | MLP · Clinical | test error rate (%): 41.77 -> 42.32 | Hurt |
| Transfer OCTree-generated features from XGBoost to MLP (ablation) | MLP · Disease | test error rate (%): 38.10 -> 35.24 | Helped |
| Transfer OCTree-generated features from XGBoost to HyperFast (ablation) | HyperFast · Clinical | test error rate (%): 43.64 -> 42.76 | Helped |
Surprising findings
- Rules remain well-optimized even without decision-tree feedback; score feedback alone improves performance, with tree reasoning giving further gains.
- The 7B Llama 2 fine-tuned on UltraChat outperformed the Llama 2 chat base and Code Llama as rule generator on datasets without descriptions.
Limitations
- Evaluating each candidate feature requires training the prediction model, which can be time-consuming for expensive models.
- Generated features are artificial and must be inspected for relevance and reliability; LLM hallucinations can yield suboptimal rules.
- Context length of the Llama 2-based generator limits use on datasets with many features.
Experimental setup
Datasets: Disease Clinical Trial Academic Enefit Tesla Stock 19 OpenML classification datasets (tabular benchmark) madelon nomao BTC Diabetes Student
Benchmarks: Disease Clinical Trial Academic Enefit Tesla Stock 19 OpenML classification datasets madelon nomao
Metrics: test error rate (%) mean absolute error relative error reduction
Baselines: XGBoost MLP HyperFast CAAFE OpenFE AutoFeat
Builds on: CAAFE OPRO CART
Closest papers by meaning
- LLM Meeting Decision Trees on Tabular Data (NeurIPS 2025 · similarity 0.90)
- Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning (ICML 2024 · similarity 0.89)
- TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks (NeurIPS 2026 · similarity 0.89)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.87)
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.87)
- UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.