Research map / Large language models / Language, safety and interpretability

UniTox: Leveraging LLMs to Curate a Unified Dataset of Drug-Induced Toxicity from FDA Labels

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses GPT-4o with a two-step chain-of-thought prompt over FDA drug labels to build UniTox, a dataset of 2,418 drugs rated for eight drug-induced toxicities, validated against FDA datasets and clinicians, and benchmarks ML toxicity classifiers on it.

Problem

Drug-induced toxicity datasets are small, limited to single organ systems, and built via time-intensive expert curation of lengthy FDA drug labels, with inconsistent methodologies across databases.

Approach

Use GPT-4o in a two-stage prompt (summarize toxicity from the full FDA label, then rate it ternary No/Less/Most or binary Yes/No from the summary) to create a unified, multi-organ toxicity dataset at scale, validated against existing FDA expert ratings and clinician review.

Key results

• On DICTrank, the full pipeline reaches 93.0% binary accuracy on ground-truth No/Most drugs versus 77.7% for askFDALabel.
• Clinician validation shows UniTox ternary ratings agree with clinicians 85–96% of the time across five toxicity types.
• Chemprop, MLP, RF and SVM models trained on the 1,349-drug small-molecule subset achieve reasonable mean ROC-AUC, but perform poorly on dermatological toxicity and ototoxicity due to class imbalance.

Measured effects reported in the paper

Surprising findings

• Removing chain-of-thought summarization consistently hurt accuracy on DICTrank, while using full label context was less effective than summaries.
• DIRIL renal toxicity validation accuracy (71.3% ternary) was notably lower than DICTrank and DILIrank, with GPT-4o often flagging renal cautions as non-toxic.

Limitations

• Only FDA-approved drugs with labels are covered, so failed drugs and trials cannot be processed.
• Ternary/binary ratings lose nuance from summaries, e.g. population-specific or animal-only toxicity.
• Clinician validation covered only 200 drugs per toxicity type from two clinicians, and DIRIL agreement was weaker.

Experimental setup

Datasets: FDALabel DICTrank DILIrank DIRIL WHO-ATC classifications PubChem

Benchmarks: DICTrank DILIrank DIRIL UniTox Small Molecule Benchmark

Metrics: accuracy concordance ROC-AUC PPV NPV

Baselines: askFDALabel GPT-3.5 Chemprop Random Forest Support Vector Machine Multilayer Perceptron

Builds on: chain-of-thought askFDALabel Chemprop

Closest papers by meaning

• Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models (NeurIPS 2024 · similarity 0.81)
• A benchmark for prediction of transcriptomic responses to chemical perturbations across cell types (NeurIPS 2024 · similarity 0.81)
• Blend the Separated: Mixture of Synergistic Experts for Data-Scarcity Drug-Target Interaction Prediction (AAAI 2025 · similarity 0.81)
• ScreenShot: A Foundation Model for Few-Shot Combination Drug Screening (NeurIPS 2026 · similarity 0.81)
• WelQrate: Defining the Gold Standard in Small Molecule Drug Discovery Benchmarking (NeurIPS 2024 · similarity 0.81)
• Soft-Label Integration for Robust Toxicity Classification (NeurIPS 2024 · similarity 0.80)