Research map / Label-efficient and robust learning / Recognition and long-tail
Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that randomized test splits for drug-target affinity prediction are dominated by high-similarity samples, and proposes Similarity Aware Evaluation (SAE), a differentiable chi-square-based split that produces test sets with desired similarity distributions.
Problem
Randomized train/test splits in drug-target binding affinity (DTA) prediction leave test sets dominated by samples highly similar to the training set, so reported performance overstates generalization to low-similarity samples encountered in practice.
Approach
Formulate test-set selection as a chi-square-constrained optimization over per-sample soft weights, relaxing the combinatorial problem with LogSumExp for the max-similarity, softmax-based soft binning, and an entropy regularizer, then solve with Lagrangian methods (PyTorch and Cooper) to obtain splits matching any desired similarity distribution.
Key results
- On EGFR IC50 with randomized split, SAM-DTA reaches overall MAE 0.6012 but MAE 1.2970 for samples with similarity below 1/3, which contain only 16 of 873 test samples.
- SAE (balanced) split gives a near-uniform test distribution across similarity bins (e.g., 309/311/308 for BACE1), and reveals lower performance on low-similarity bins.
- In the mimic split on EGFR, SAE-selected hyperparameters yield external-test MAE of 0.9773 for SAM-DTA, the best among split strategies compared (random 0.9863).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate SAM-DTA on low-similarity rather than high-similarity test samples | SAM-DTA · EGFR IC50 randomized split | MAE: 0.5743 -> 1.6787 | Hurt |
| Select hyperparameters using SAE mimic split instead of randomized split | SAM-DTA · EGFR IC50 external test | MAE: 0.9863 -> 0.9773 | Helped |
| Select hyperparameters using SAE mimic split instead of randomized split | SAM-DTA · EGFR IC50 external test | R^2: 0.3206 -> 0.3367 | Helped |
| Use SAE mimic split rather than other split strategies for hyperparameter selection | Five DTA prediction methods · EGFR IC50 external test | MAE | Mixed |
| Use SAE balanced split instead of randomized split | SAM-DTA · BACE1 IC50 | MAE: 0.4834 -> 0.7272 | Hurt |
| Use SAE balanced split instead of randomized split | SAM-DTA · Carbonic anhydrase I Ki | MAE: 0.5323 -> 0.7031 | Hurt |
| Use SAE balanced split instead of randomized split | SAM-DTA · Carbonic anhydrase II Ki | MAE: 0.5785 -> 0.7738 | Hurt |
Surprising findings
- Low-similarity samples cause large error increases (e.g., R^2 turns negative) yet barely affect the overall randomized-split metric because they are a tiny fraction of the test set.
- Under the mimic split, SAE gave worse performance on the high-similarity bin in external tests, which the authors attribute to fewer high-similarity internal samples.
Limitations
- Approximation error from relaxing binary split weights and the LogSumExp/softmax surrogates, requiring regularization and hyperparameter tuning (beta, sigma, lambda).
- Quadratic time and space complexity in the number of samples, motivating sparse approximations for large libraries.
- Evaluation is limited to four datasets and five DTA methods, and the split depends on the choice of fingerprint, similarity measure, and aggregation.
Experimental setup
Datasets: BindingDB EGFR IC50 BindingDB BACE1 IC50 BindingDB Carbonic anhydrase I Ki BindingDB Carbonic anhydrase II Ki ChEMBL PubChem
Benchmarks: EGFR IC50 (BindingDB) BACE1 IC50 Carbonic anhydrase I Ki Carbonic anhydrase II Ki
Metrics: MAE R^2
Baselines: Random split Scaffold split SIMPD split Stratified split Dissimilar split
Builds on: Cooper MolCLR SAM-DTA SIMPD
Closest papers by meaning
- Towards precision protein-ligand affinity prediction benchmark: A Complete and Modification-Aware DAVIS Dataset (NeurIPS 2025 · similarity 0.86)
- Drug-TTA: Test-Time Adaptation for Drug Virtual Screening via Multi-task Meta-Auxiliary Learning (ICML 2025 · similarity 0.86)
- Blend the Separated: Mixture of Synergistic Experts for Data-Scarcity Drug-Target Interaction Prediction (AAAI 2025 · similarity 0.85)
- UniMatch: Universal Matching from Atom to Task for Few-Shot Drug Discovery (ICLR 2025 · similarity 0.85)
- Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization (NeurIPS 2024 · similarity 0.85)
- Enhancing Ligand Validity and Affinity in Structure-Based Drug Design with Multi-Reward Optimization (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.