Research map / Large language models / Retrieval and knowledge

Small Models are LLM Knowledge Triggers for Medical Tabular Prediction

ICLR 2025 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SERSAL, an unsupervised self-prompting loop where a blackbox LLM's soft zero-shot confidences train a small FT-Transformer via adapted DivideMix, and the small model's soft labels then fine-tune the LLM, improving GPT-3.5/GPT-4 tabular medical prediction without gold labels.

Problem

Blackbox LLMs lag far behind supervised small models on numeric tabular prediction (e.g., medical diagnosis from lab values), and standard linguistic prompting yields little gain on such data.

Approach

Use the LLM's per-sample confidence as noisy soft labels to teach a small tabular model with a semi-supervised noisy-label method (adapted DivideMix, early stopping on high-confidence samples), then reversely fine-tune the LLM on the small model's sharpened predictions, iterating the loop.

Key results

• SERSAL with GPT-3.5 (single loop) reaches 91.39 AUC on HF vs 71.88 for 0-shot GPT-3.5 and 88.19 for supervised FT-Transformer, without gold labels.
• SERSAL with GPT-4 averages higher AUC than supervised FT-Transformer on several medical datasets, e.g., 94.18 on HF and 92.68 on ECD.
• Three SERSAL loops raise GPT-3.5 LI AUC from 79.39 to 84.07 and ECD from 86.40 to 89.00.

Measured effects reported in the paper

Surprising findings

• Standard prompting tricks (8-shot, CoT) give little or negative gains on tabular medical data, unlike on textual tasks.
• On a randomly generated 'Fake' dataset with no domain knowledge, SERSAL degrades AUC (46.28 0-shot to 38.72), showing it cannot correct confidently wrong LLM knowledge.

Limitations

• Requires the LLM to already hold relevant domain knowledge; fails on unknown domains such as the Fake dataset.
• Evaluated only on binary classification datasets with a limited set of LLMs (GPT-3.5, GPT-4).
• Relies on the LLM being fine-tunable via API, so it is not applicable to open-weight usage without fine-tuning access.

Experimental setup

Datasets: Heart Failure Prediction Lung Cancer Prediction Early Classification of Diabetes Indian Liver Patient Records Hepatitis C Prediction Pima Indians Diabetes Database Framingham Heart Study Stroke Prediction COVID-19 Presence Anemia Disease Churn Modeling Credit

Benchmarks: Heart Failure Prediction Lung Cancer Prediction Early Classification of Diabetes Indian Liver Patient Records Hepatitis C Prediction Pima Indians Diabetes Database Framingham Heart Study Stroke Prediction COVID-19 Presence Anemia Disease Churn Modeling Credit

Metrics: AUC

Baselines: Zero-shot prompting Zero-shot CoT 8-shot prompting TabLLM LIFT FT-Transformer (supervised)

Builds on: DivideMix FT-Transformer zero-shot CoT TabLLM LIFT

Closest papers by meaning

• LLM Meeting Decision Trees on Tabular Data (NeurIPS 2025 · similarity 0.88)
• LLMs can construct powerful representations and streamline sample-efficient supervised learning (NeurIPS 2026 · similarity 0.87)
• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.87)
• Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.87)
• Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.86)
• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.86)