Research map / Label-efficient and robust learning / Self-supervised and contrastive
3D Denoisers Are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Denoise and Distill (D&D), pretraining a 3D TorchMD-NET teacher via conformer denoising and distilling its representations into a 2D TokenGT graph student for molecular property prediction without conformers at downstream time.
Problem
Molecular property prediction needs pretraining that transfers 3D knowledge to 2D graph encoders, since 3D denoising pretraining requires accurate conformers at downstream finetuning, which are expensive to compute at scale.
Approach
Combine denoising pretraining of a 3D conformer encoder with cross-modal knowledge distillation into a 2D graph encoder (graph-level or node-level alignment), so the 2D student can be finetuned on graphs alone while inheriting force-field-like knowledge.
Key results
- D&D-Node (+mp) achieves best LIPO RMSE 0.6975 vs 0.9720 for RandInit, and best BP and MP MAE (0.2220 and 0.2937) among compared methods.
- On QM9, D&D-Node reaches alpha MAE 0.2807 versus 0.3268 for 3DInfomax and 0.3972 for RandInit.
- The method is reported to beat RandInit on 11 of 12 OGB tasks, with average gains of 8.8% on OGB and 36.9% on curated datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain with 3D denoising and distill its representations to a 2D graph encoder | D&D versus RandInit · OGB and Curated molecular property tasks | performance: 8.8%; 36.9% | Helped |
| Use 3D denoising plus distillation instead of contrastive 3DInfomax pretraining | D&D · OGB and Curated molecular property tasks | performance: 15.8%; 37.2% | Helped |
| Distill node-level rather than graph-pooled representations (ablation) | D&D · QM9 quantum properties | MAE | Mixed |
| Distill graph-pooled rather than node-level representations (ablation) | D&D · OGB and Curated molecular property tasks | performance | Mixed |
| Use mean pooling instead of a virtual-node representation (ablation) | D&D-Graph and D&D-Node · OGB and Curated molecular property tasks | performance | Mixed |
| Use a virtual-node representation instead of mean pooling (ablation) | D&D-Graph and D&D-Node · OGB and Curated molecular property tasks | performance | Mixed |
Surprising findings
- Distilling graph-level mean-pooled representations (D&D-Graph) yields lower training loss but a larger generalization gap than node-level distillation.
- Contrastive 3DInfomax gives only marginal gains over random initialization on many OGB tasks.
Limitations
- Only TokenGT (attention-based) explored as 2D student in main experiments; message-passing GNNs not tested.
- Gains are limited on BACE and SIDER, possibly due to molecule size mismatch with PCQM4Mv2.
- No principled way to choose between D&D-Graph and D&D-Node variants; requires tuning.
Experimental setup
Datasets: PCQM4Mv2 ESOL FREESOLV LIPO BACE BBBP CLINTOX HIV SIDER TOX21 TOXCAST QM9
Benchmarks: OGB molecular property prediction (ESOL, FREESOLV, LIPO, BACE, BBBP, CLINTOX, HIV, SIDER, TOX21, TOXCAST) curated BP, MP, RI, logP QM9
Metrics: RMSE ROC-AUC MAE
Baselines: RandInit 3DInfomax GraphCL ProPred DisPred ConfGen
Builds on: 3D denoising pretraining (Zaidi et al.) TorchMD-NET TokenGT 3DInfomax structured knowledge distillation
Closest papers by meaning
- Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling (ICML 2025 · similarity 0.90)
- SubgDiff: A Subgraph Diffusion Model to Improve Molecular Representation Learning (NeurIPS 2024 · similarity 0.90)
- Enhancing Molecular Representation Learning Through the Combination of 3D and 2D Graph Machine Learning (Student Abstract) (AAAI 2025 · similarity 0.89)
- Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks (ICLR 2024 · similarity 0.88)
- Empowering Active Learning for 3D Molecular Graphs with Geometric Graph Isomorphism (NeurIPS 2024 · similarity 0.88)
- 3D-GSRD: 3D Molecular Graph Auto-Encoder with Selective Re-mask Decoding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.