Research map / Large language models / Language, safety and interpretability

Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes

ICML 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLLM, which prompts a frozen LLM (GPT-4/GPT-3.5) with dataset context and examples to generate synthetic tabular data for low-data regimes, then curates samples using XGBoost learning dynamics (confidence and aleatoric uncertainty).

Problem

Tabular data augmentation when the labeled training set is very small (n<100), where conventional tabular generators (CTGAN, TVAE, TabDDPM, SMOTE) and LLM fine-tuning struggle to produce diverse, accurate synthetic data.

Approach

Use in-context generation from a frozen black-box LLM with feature descriptions, then filter synthetic samples via learning dynamics (average confidence and aleatoric uncertainty across checkpoints of a classifier trained on the small real set), discarding low-confidence, low-aleatoric samples likely to be mislabeled.

Key results

• CLLM with GPT-4 plus curation achieves the best average rank across sample sizes (2.71 at n=20, 2.14 at n=40), outperforming baselines in 20 of 28 dataset/size settings.
• Curation improves downstream AUC for nearly all generators; e.g., on Adult (n=20), GPT-4 AUC rises from 50.39 uncurated to 71.48 curated.
• Downstream AUC on synthetic-data-trained models is strongly negatively linearly related (slope about -1.4) to the fraction of synthetic samples discarded by curation.

Measured effects reported in the paper

Surprising findings

• Uncurated GPT-4 data on Adult performs very poorly (AUC about 50 at n=20), far below real-data baselines, showing LLM generation can be unreliable without curation.
• Fine-tuning an LLM (GReaT) underperforms in-context generation at low sample sizes, with overfitting cited as the cause.

Limitations

• Performance is tied to LLM backbone size; smaller open-source LLMs yield poorer uncurated data.
• Private medical datasets cannot be released, and GPT-4/GPT-3.5 results depend on closed API access.
• Curation thresholds and XGBoost curation model are heuristically chosen; biases in LLM outputs are not addressed by curation.

Experimental setup

Datasets: Adult Income Compas Covid-19 (private) CUTRACT Prostate (private) Drug MAGGIC (private) SEER Prostate (private)

Benchmarks: Covid-19 (private) CUTRACT (private) MAGGIC (private) SEER (private) Compas Adult Drug

Metrics: AUC average rank precision recall

Baselines: CTGAN TVAE TabDDPM NFLOW SMOTE GReaT GPT-3.5

Builds on: GReaT TabDDPM CTGAN SMOTE TabPFN

Closest papers by meaning

• Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.88)
• Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning (ICML 2024 · similarity 0.87)
• It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.87)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
• Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.87)
• DataComp-LM: In search of the next generation of training sets for language models (NeurIPS 2024 · similarity 0.87)