Research map / Graph learning / Graph self-supervised learning
3D-GSRD: 3D Molecular Graph Auto-Encoder with Selective Re-mask Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-GSRD, a 3D masked graph auto-encoder that re-masks only 3D-relevant encoder features, adds a distilled 2D graph position encoder, and uses a structure-independent Transformer decoder, with a new 3D Relational-Transformer encoder.
Problem
Extending re-mask decoding from 2D to 3D masked graph modeling is hard: decoders that see 2D bond structure leak structure and weaken the encoder, while structure-free decoders cannot locate re-masked atoms in the 2D graph.
Approach
Selective re-mask decoding re-masks only 3D-relevant encoder features while keeping 2D context via a stop-gradient 2D graph position encoder distilled (cosine loss) from the 3D encoder, paired with a structure-independent Transformer decoder and a 3D Relational-Transformer encoder.
Key results
- 3D-GSRD achieves state-of-the-art on 7 of 8 MD17 molecules (e.g., Aspirin MAE 0.0583 vs 0.0726 for 3D-ReTrans and 0.0825 for Frad), except Benzene.
- On QM9, 3D-GSRD sets a new state-of-the-art on 7 of 12 properties, e.g., mu 0.009 D and U0 5.48 meV.
- Ablation: adding SRD with distillation lowers Toluene MAE from 0.0291 (structure-dependent decoder) to 0.0275 on MD17 for the structure-independent variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3D-GSRD pretraining instead of Frad | 3D-ReTrans · MD17 Aspirin | force MAE: 0.0825 -> 0.0583 | Helped |
| Use 3D-GSRD pretraining instead of Frad | 3D-ReTrans · MD17 Benzene | force MAE: 0.1355 -> 0.1435 | Hurt |
| Use 3D-GSRD pretraining instead of Frad | 3D-ReTrans · MD17 Salicylic | force MAE: 0.0569 -> 0.0356 | Helped |
| Replace the structure-dependent decoder with a structure-independent decoder (ablation) | 3D-GSRD pretraining · MD17 Toluene | MAE: 0.0291 -> 0.0275 | Helped |
| Remove SRD and distillation from the structure-independent decoder (ablation) | 3D-GSRD pretraining · MD17 Salicylic | MAE: 0.0387 -> 0.0404 | Hurt |
| Add SRD without distillation (ablation) | Structure-independent decoder · MD17 Salicylic | MAE: 0.0404 -> 0.0416 | Hurt |
| Add 2D-from-3D distillation (ablation) | Structure-independent decoder with SRD · MD17 Salicylic | MAE: 0.0416 -> 0.0387 | Helped |
| Add 3D data augmentation (ablation) | Relational-Transformer · QM9 homo | MAE: 27.7 -> 24.6 | Helped |
| Add 3D Relational-Attention (ablation) | Relational-Transformer with 3D data augmentation · QM9 homo | MAE: 24.6 -> 23.4 | Helped |
| Add the 3D Update Layer (ablation) | 3D Relational-Attention encoder · QM9 homo | MAE: 23.4 -> 22.0 | Helped |
| Replace 2D-PE with RWSE (ablation) | 3D-GSRD · MD17 Salicylic | MAE: 0.0356 -> 0.0368 | Hurt |
| Replace 2D-PE with RWSE (ablation) | 3D-GSRD · MD17 Uracil | MAE: 0.0292 -> 0.0310 | Hurt |
Surprising findings
- Fusing 2D-PE directly into downstream prediction without distillation consistently underperforms, while with distillation the explicit 2D input becomes unnecessary.
- Using 2D-PE alone without distillation degrades downstream performance, attributed to 2D structure leakage into the decoder.
Limitations
- Pretraining uses only 3.37M molecules from PCQM4Mv2, much smaller than datasets like PubChemQC.
- No error bars or repeated runs due to limited compute.
- Focused on molecular property prediction; generation and text-multimodal tasks untested.
Experimental setup
Datasets: PCQM4Mv2 QM9 MD17
Benchmarks: MD17 QM9
Metrics: MAE
Baselines: TorchMD-NET 3D-EMGP Frad Transformer-M SE(3)-DDM Coordinate Denoising SliDe Mol-AE Uni-Mol2 ComENet SphereNet PaiNN
Builds on: Relational-Transformer TorchMD-NET GraphMVP Transformer AlphaFold3
Closest papers by meaning
- 3D Denoisers Are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation (AAAI 2025 · similarity 0.88)
- Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling (ICML 2025 · similarity 0.87)
- Mol-AE: Auto-Encoder Based Molecular Representation Learning With 3D Cloze Test Objective (ICML 2024 · similarity 0.86)
- Empowering Active Learning for 3D Molecular Graphs with Geometric Graph Isomorphism (NeurIPS 2024 · similarity 0.86)
- Enhancing Molecular Representation Learning Through the Combination of 3D and 2D Graph Machine Learning (Student Abstract) (AAAI 2025 · similarity 0.86)
- 3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.