Research map / Large language models / Instruction and fine-tuning

LLM Meeting Decision Trees on Tabular Data

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeLTa, which uses GPT-4o to refine a set of random-forest decision tree rules into one new partition rule, then learns per-leaf gradient-correction models that adjust random-forest predictions, achieving LLM-based tabular SOTA without serialization or LLM fine-tuning.

Problem

LLM-based tabular prediction methods serialize tabular rows into text (raising privacy and feature-name dependence issues) and either fine-tune LLMs or rely on in-context learning limited to few-shot or classification settings.

Approach

Use an LLM only to read a set of decision tree rules from a random forest and synthesize a refined rule that partitions the training space into more coherent leaves; then fit per-leaf CART/TabPFN models to the negative gradients of the forest's loss and add the sample-specific correction to the forest prediction.

Key results

• On full-data classification, DeLTa averages 0.822 accuracy versus 0.801 for the best LLM baseline (TabLLM) across the six classification datasets.
• With 64 training samples, DeLTa averages 0.660 accuracy versus 0.628 for FeatLLM.
• DeLTa avoids per-sample LLM queries: training takes 35.2 s on Adult with full data versus 1231 s for FeatLLM, and inference takes 0.09 s versus 90149 s for LIFT.

Measured effects reported in the paper

Surprising findings

• Swapping GPT-4o for the open-weight Qwen3-32B leaves average performance nearly unchanged (0.816 vs 0.822 full-data classification).
• Ablation shows that without rule refinement (RR), accuracy on CR drops from 0.7829 to 0.7257, suggesting the LLM-refined rule itself matters.

Limitations

• Relies on proprietary GPT-4o for the main results, though a Qwen3-32B variant is tested.
• Reported gains on some datasets are small or absent, e.g., Car and BA where TabLLM or TP-BERTa is comparable or better.
• Number of leaves for the LLM-generated rule is capped below 30, limiting the partition granularity.

Experimental setup

Datasets: Blood Credit Car Bank Adult Jannis Cpu_act Credit_reg California_housing House_16H Fried Diamond

Benchmarks: Blood Credit Car Bank Adult Jannis Cpu_act Credit_reg California_housing House_16H Fried Diamond

Metrics: accuracy NRMSE

Baselines: TabLLM LIFT LIFT-ICL TP-BERTa GTL P2T FeatLLM KNN CART MLP Random Forest XGBoost

Builds on: Random Forest CART Gradient Boosting TabLLM LIFT

Closest papers by meaning

• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning (NeurIPS 2024 · similarity 0.90)
• Small Models are LLM Knowledge Triggers for Medical Tabular Prediction (ICLR 2025 · similarity 0.88)
• Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.87)
• Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning (ICML 2024 · similarity 0.87)
• Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes (ICML 2024 · similarity 0.87)
• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.86)