Research map / Vision-language and multimodal / Human and embodied modeling
PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PyTDC, an open-source platform for training, evaluating and inferring with multimodal biomedical/single-cell foundation models, with a model server, API-first data retrieval, and a new single-cell drug-target nomination benchmark on RA and IBD.
Problem
Existing biomedical benchmarks lack end-to-end infrastructure for training, evaluation, and inference of models integrating multimodal biological data across therapeutic ML tasks, and context-specific (cell-type-specific) evaluation for drug-target nomination is underexplored.
Approach
An API-first platform that unifies heterogeneous continuously updated data sources and model weights through a model server with standardized inference endpoints, plus context-specific metrics (top-K cell-type AUROC and AP@R) for single-cell drug-target nomination.
Key results
- On RA drug-target nomination, PINNACLE reaches AP@5 Top-20 CT of 0.917 vs 0.585 for Node2Vec, 0.334 for GATv2 and 0.105 for label propagation.
- GATv2 yields AUROC of 0.5 at all top-K cell-type settings, i.e. no contextualized signal, while PINNACLE reaches 0.765 AUROC Top-1 CT for RA.
- On unseen genetic perturbations (Norman K562), GEARS reduces MSE@20DEG to 0.176 vs 0.230 for CPA and 0.341 for no-perturb.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cell-type-specific PINNACLE rather than GATv2 | Drug-target nomination · RA, AP@5 Top-20 CT | AP@5: 0.334\std 0.050 -> 0.917\std 0.057 | Helped |
| Use cell-type-specific PINNACLE rather than GATv2 | Drug-target nomination · IBD, AP@5 Top-20 CT | AP@5: 0.348\std 0.043 -> 0.884\std 0.071 | Helped |
| Use cell-type-specific PINNACLE rather than Node2Vec | Drug-target nomination · RA, AP@5 Top-20 CT | AP@5: 0.585\std 0.045 -> 0.917\std 0.057 | Helped |
| Use cell-type-specific PINNACLE rather than Node2Vec | Drug-target nomination · IBD, AP@5 Top-20 CT | AP@5: 0.617\std 0.081 -> 0.884\std 0.071 | Helped |
| Use Node2Vec rather than PINNACLE | Drug-target nomination · RA, AUROC Top-1 CT | AUROC: 0.765\std 0.054 -> 0.766 \std 0.030 | No clear effect |
| Use GEARS rather than CPA | Genetic perturbation response prediction · Norman K562 unseen perturbations | MSE@20DEG: 0.230\std 0.008 -> 0.176\std 0.003 | Helped |
| Use GEARS rather than CPA | Genetic perturbation response prediction · Replogle 562 unseen perturbations | MSE@20DEG: 0.126\std 0.000 -> 0.109\std 0.004 | Helped |
| Use GEARS rather than CPA | Genetic perturbation response prediction · Replogle RPE1 unseen perturbations | MSE@20DEG: 0.162\std 0.001 -> 0.110\std 0.003 | Helped |
| Evaluate ChemCPA on unseen rather than seen perturbations | ChemCPA · BMS | R²: 0.943\std 0.006 -> 0.906\std 0.006 | Hurt |
| Evaluate ChemCPA on unseen rather than seen perturbations | ChemCPA · Dex | R²: 0.882\std 0.014 -> 0.540\std 0.013 | Hurt |
| Evaluate ChemCPA on unseen rather than seen perturbations | ChemCPA · Nutlin | R²: 0.925\std 0.010 -> 0.835\std 0.009 | Hurt |
| Evaluate ChemCPA on unseen rather than seen perturbations | ChemCPA · SAHA | R²: 0.825\std 0.026 -> 0.690\std 0.021 | Hurt |
Surprising findings
- Label propagation performs worse than random on RA context-specific AUROC (0.451 at Top-1 CT and lower at larger K), and GATv2 reaches only chance-level AUROC.
- ChemCPA performance dropped by an average of 15% on unseen chemical perturbations, with a maximum drop of 34%.
Limitations
- The best model (PINNACLE) is engineered for cell type-specific PPI networks and only works with a priori-defined contexts, and does not generalize to unseen cell types.
- Models cannot incorporate additional modalities in the reported case study.
- Licensing of released models and datasets cannot be strictly enforced by PyTDC.
Experimental setup
Datasets: Open Targets Platform DrugBank scPerturb sciPlex2 Norman K562 Replogle K562 Replogle RPE1
Benchmarks: tdc_ml.scDTN (RA and IBD) scPerturb genetic perturbation benchmark sciPlex2 chemical perturbation benchmark
Metrics: AP@5 AP@20 AUROC at top-K cell types MSE@20DEG R^2
Baselines: PINNACLE GATv2 Node2Vec Label Propagation chemCPA Biolord scGen CPA
Builds on: Therapeutic Data Commons PINNACLE GATv2 node2vec GEARS
Closest papers by meaning
- Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine (ICLR 2025 · similarity 0.85)
- BioXArena: Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks (NeurIPS 2026 · similarity 0.84)
- When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework (NeurIPS 2026 · similarity 0.84)
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.83)
- On the Scalability of GNNs for Molecular Graphs (NeurIPS 2024 · similarity 0.83)
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.