Research map / Large language models / Retrieval and knowledge
A Dataset for Distilling Knowledge Priors from Literature for Therapeutic Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Medex, a 32.3M-fact dataset distilled from literature via LLM pipelines linking therapeutic entities (SMILES/protein IDs) to natural-language facts, and shows small contrastively trained models on it match or beat larger TxGemma models on TDC tasks.
Problem
Therapeutic design simulators and benchmarks (e.g., GuacaMol) ignore practical constraints like safety and ADME, and the experimental knowledge needed to encode them is locked in unstructured literature.
Approach
Extract entity-fact pairs from millions of paragraphs using distilled LLMs (Llama 3.1 8B for entity tagging, Gemma 3 4B for fact extraction) and use them as contrastive pretraining for small entity-text encoders (MedexCLIP), then reuse the embeddings as supervised heads and as safety constraints in Bayesian optimization.
Key results
- MedexCLIP (15M trainable parameters) achieves average classification score 0.771 vs. 0.768 for TxGemma-2B and reduces average regression MAE by 33% relative to TxGemma-2B, beating it on 23/28 regression tasks.
- In zero-shot evaluation on 9 design-relevant endpoints, MedexCLIP reaches mean AUROC 0.718 vs. 0.411 for Gemma-2B (74% relative improvement).
- Over 60% of molecules proposed on GuacaMol by evaluated models were classified as having high mutagenicity probability; constrained Bayesian optimization finds feasible molecules with scores close to unconstrained BO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain on Medex literature facts and use a CLIP-style structure-text model | MedexCLIP · TDC classification tasks | mean score: 0.768 to 0.771 | Helped |
| pretrain on Medex literature facts and use a CLIP-style structure-text model | MedexCLIP · TDC regression tasks | average MAE: 33% lower MAE | Helped |
| use MedexCLIP zero-shot classification rather than Gemma-2B | TDC safety and ADME endpoints · 9 selected TDC endpoints | mean AUROC: 0.411 to 0.718; 74% relative improvement | Helped |
| use CLIP-style structure-text architecture instead of soft-prompted or text-only models (ablation) | Medex models · TDC toxicity classification (8 tasks) | AUROC: 0.800 to 0.837 | Helped |
| use CLIP-style structure-text architecture instead of soft-prompted or text-only models (ablation) | Medex models · TDC toxicity classification (2 tasks) | Accuracy: 0.798 to 0.807 | Helped |
| use CLIP-style structure-text architecture instead of soft-prompted or text-only models (ablation) | Medex models · TDC pharmacokinetics classification (13 tasks) | AUROC: 0.781 to 0.830 | Mixed |
| use CLIP-style structure-text architecture instead of soft-prompted or text-only models (ablation) | Medex models · TDC high-throughput screening (4 tasks) | AUROC: 0.711 to 0.737 | Helped |
| use Medex-derived knowledge rather than TDC-only training (ablation) | MedexCLIP · TDC clinical trial outcome classification (3 tasks) | AUROC: 0.683 to 0.631 | Hurt |
Surprising findings
- Over 60% of GuacaMol-proposed molecules had high predicted mutagenicity probability under supervised classifiers.
- No proposals met the criteria when requiring both non-toxicity and favorable ADME at 95% certainty.
Limitations
- Facts are not weighted by provenance or experimental quality; each excerpt is treated as a source of truth.
- Fact corpus is a subset (~12M paragraphs) of the full literature; does not differentiate higher- and lower-certainty facts.
- Dataset inherits biases of biomedical literature, including over-representation of well-studied molecules and positive-result bias.
Experimental setup
Datasets: Medex TDC PubChem ZINC20 PubTator3
Benchmarks: TDC GuacaMol
Metrics: AUROC AUPRC MAE PCC Spearman accuracy
Baselines: TxGemma 2B TDC LM (Qwen 2.5 0.5B fine-tuned on TDC) Gemma 2 2B zero-shot Specialist SOTA models
Builds on: CLIP LLaVA TxGemma LOLBO SCBO
Closest papers by meaning
- Knowledge Enhanced Representation Learning for Drug Discovery (AAAI 2024 · similarity 0.85)
- Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models (NeurIPS 2024 · similarity 0.85)
- BioXArena: Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks (NeurIPS 2026 · similarity 0.84)
- BounDr.E: Predicting Drug-likeness via Biomedical Knowledge Alignment and EM-like One-Class Boundary Optimization (ICML 2025 · similarity 0.84)
- Generating Novel Leads for Drug Discovery Using LLMs with Logical Feedback (AAAI 2024 · similarity 0.83)
- From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.