Research map / Optimization / Optimizers
Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GraphSAM, an efficient sharpness-aware minimization variant for molecular graph transformers that approximates the perturbation gradient with a moving average of previous updating gradients, periodically re-anchored, plus a rho scheduler.
Problem
SAM doubles training cost because it needs two forward/backward passes per step, and existing efficient SAM variants fail to preserve generalization for graph transformers on molecular property prediction without pre-training.
Approach
Approximate the SAM perturbation gradient at step t+1 by an exponential moving average of the previous perturbation gradient and the normalized previous updating gradient, re-anchoring it with an exact computation periodically, and schedule the perturbation radius rho by a StepLR-like decay, so each step needs only one forward/backward pass in most iterations.
Key results
- GraphSAM on CoMPT reaches 0.961 ROC-AUC on BBBP and 0.608 RMSE on Lipophilicity, close to SAM (0.962 and 0.611) while running at 155.4% of SAM's training throughput on BBBP.
- GraphSAM improves GROVER from 0.917 (Adam) to 0.928 ROC-AUC on BBBP, versus 0.926 for SAM.
- Efficient variants such as SAM-One drop CoMPT BBBP ROC-AUC to 0.922 and RST to 0.940, below GraphSAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GraphSAM instead of SAM | GROVER · BBBP | ROC-AUC: 0.926 -> 0.928 | Helped |
| Use GraphSAM instead of SAM | GROVER · Tox21 | ROC-AUC: 0.840 -> 0.846 | Helped |
| Use GraphSAM instead of SAM | GROVER · ClinTox | ROC-AUC: 0.872 -> 0.866 | Hurt |
| Use GraphSAM instead of SAM | GROVER · ESOL | RMSE: 0.619 -> 0.625 | Hurt |
| Use GraphSAM instead of SAM | CoMPT · ClinTox | ROC-AUC: 0.927 -> 0.937 | Helped |
| Use GraphSAM instead of SAM | CoMPT · ESOL | RMSE: 0.517 -> 0.511 | Helped |
| Use GraphSAM instead of SAM | CoMPT · BBBP | ROC-AUC: 0.962 -> 0.961 | Hurt |
| Use GraphSAM instead of SAM | GROVER · BACE | ROC-AUC: 0.886 -> 0.882 | Hurt |
| Use GraphSAM instead of SAM | GROVER · QM7 | MAE: 76.4 -> 75.8 | Helped |
| Use GraphSAM instead of SAM | CoMPT · QM8 | MAE: 0.0145 -> 0.0141 | Helped |
| Set moving-average β to 0.9 rather than 0.99 (ablation) | CoMPT GraphSAM · BBBP | ROC-AUC: 0.961 ± 0.012 -> 0.958 ± 0.014 | Hurt |
| Set moving-average β to 0.999 rather than 0.99 (ablation) | CoMPT GraphSAM · BBBP | ROC-AUC: 0.961 ± 0.012 -> 0.959 ± 0.003 | Hurt |
Surprising findings
- Reusing a single perturbation gradient for all steps (SAM-One) yields the worst generalization among the compared variants, despite the claimed smooth change of the perturbation gradient.
- Existing efficient SAM variants (LookSAM, AE-SAM, RST) sometimes perform no better than Adam on molecular graph models, indicating that vision-domain efficiency tricks do not transfer.
Limitations
- Theoretical conjectures are supported mainly by first-order Taylor arguments and empirical observations rather than rigorous bounds; some claimed proofs are informal.
- Gains over SAM are often within standard deviations, and GraphSAM's throughput is lower than the one-pass ideal (about 1.2-1.5x SAM speed, not 2x).
- Performance depends on re-anchoring frequency and hyperparameters (beta, rho, lambda, gamma) tuned per dataset.
Experimental setup
Datasets: BBBP Tox21 Sider ClinTox ESOL Lipophilicity BACE QM7 QM8
Benchmarks: BBBP Tox21 Sider ClinTox ESOL Lipophilicity BACE QM7 QM8
Metrics: ROC-AUC RMSE MAE throughput (graphs/s)
Baselines: Adam SAM SAM-One SAM-k LookSAM AE-SAM RST GCN MPNN DMPNN CMPNN
Builds on: SAM LookSAM RST AE-SAM
Closest papers by meaning
- Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.87)
- Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification (NeurIPS 2024 · similarity 0.87)
- Generalizable Insights for Graph Transformers in Theory and Practice (NeurIPS 2025 · similarity 0.86)
- GraphGPT: Generative Pre-trained Graph Eulerian Transformer (ICML 2025 · similarity 0.85)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.85)
- Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.