Research map / Label-efficient and robust learning / Recognition and long-tail
RoFt-Mol: Benchmarking Robust Fine-tuning with Molecular Graph Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks eight robust fine-tuning methods (full-FT, LP, surgical FT, LP-FT, WiSE-FT, L2-SP, feature-map, BSS) on molecular graph foundation models across classification/regression, few-shot and OOD splits, and proposes DWiSE-FT, a layer-wise weight-interpolation variant.
Problem
Fine-tuning molecular graph foundation models is hard due to small pre-training datasets, scarce labeled downstream data, and mixed regression/classification objectives; it is unclear which robust fine-tuning method works best under these conditions.
Approach
Organizes eight fine-tuning methods into weight-based, representation-based, and partial fine-tuning mechanisms and benchmarks them across pre-training paradigms, task types, label sizes and splits; then extends WiSE-FT to layer-wise mixing coefficients (DWiSE-FT) optimized via validation-loss gradients, inspired by L2-SP's implicit per-weight mixing.
Key results
- Under few-shot classification with Mole-BERT, WiSE-FT is among the strongest weight-based methods, while L2-SP leads on regression; Feature-map and Surgical-FT lead for supervised Graphium-Toy in many settings.
- Partial fine-tuning (surgical FT, LP-FT) improves over full-FT in non-few-shot settings but ranks worst in few-shot settings.
- DWiSE-FT consistently outperforms WiSE-FT on few-shot regression (Fewshot 50/100) and often matches or surpasses L2-SP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use layer-wise rather than a single fixed weight-interpolation coefficient | Mole-BERT fine-tuning · Fewshot regression, random and size splits | RMSE: 1.384 ± 0.047 -> 1.378 ± 0.055; 2.615 ± 0.072 -> 1.488 ± 0.101 | Helped |
| Use layer-wise rather than a single fixed weight-interpolation coefficient | Mole-BERT fine-tuning · Fewshot regression, scaffold split | RMSE: 1.842 ± 0.056 -> 1.616 ± 0.047; 1.177 ± 0.009 -> 1.110 ± 0.013; 1.162 ± 0.004 -> 1.173 ± 0.005; 2.454 ± 0.043 -> 2.306 ± | Mixed |
| Train only the prediction head instead of fine-tuning the encoder | Self-supervised pre-trained models · Downstream molecular tasks, all splits | AUC or RMSE | Hurt |
| Use partial fine-tuning rather than full fine-tuning | Self-supervised pre-trained models · Fewshot molecular tasks, especially regression | AUC or RMSE | Hurt |
| Use weight-based fine-tuning rather than other strategies | Self-supervised pre-trained models · Fewshot classification | AUC | Helped |
| Use weight-based fine-tuning rather than other strategies | Self-supervised pre-trained models · Fewshot regression | RMSE | Helped |
| Preserve pre-trained representations through representation regularization | Supervised pre-trained models · Downstream molecular tasks | AUC or RMSE | Helped |
| Use supervised instead of self-supervised pre-training | Molecular graph foundation models · Fewshot downstream tasks | AUC or RMSE | Helped |
| Use supervised instead of self-supervised pre-training | Molecular graph foundation models · Non-fewshot tasks without pre-training task alignment | AUC or RMSE | Mixed |
Surprising findings
- Linear probing is consistently the worst method for self-supervised pre-trained molecular models, contrary to CV findings where LP is robust OOD.
- Partial fine-tuning underfits in few-shot settings, and the effect is more severe for regression tasks.
Limitations
- Gains of DWiSE-FT are shown only for few-shot regression in the main text; the classification and full-data results are less developed.
- Only a limited set of molecular pre-trained models (six) and eight fine-tuning methods are evaluated.
Experimental setup
Datasets: ClinTox BBBP BACE HIV MUV Sider Tox21 ToxCast ESOL Lipo Malaria CEP
Benchmarks: ClinTox BBBP BACE HIV MUV Sider Tox21 ToxCast ESOL Lipo Malaria CEP
Metrics: AUC RMSE average rank
Baselines: Full-FT Linear probing Surgical-FT LP-FT WiSE-FT L2-SP Feature-map BSS
Builds on: WiSE-FT L2-SP LP-FT Surgical fine-tuning BSS
Closest papers by meaning
- Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning (ICML 2024 · similarity 0.85)
- MORE: Molecule Pretraining with Multi-Level Pretext Task (AAAI 2025 · similarity 0.85)
- Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets (ICLR 2024 · similarity 0.85)
- On the Scalability of GNNs for Molecular Graphs (NeurIPS 2024 · similarity 0.85)
- MIPT: Multilevel Informed Prompt Tuning for Robust Molecular Property Prediction (ICML 2025 · similarity 0.85)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.