Research map / 3D vision / 3D generation
Mol-AE: Auto-Encoder Based Molecular Representation Learning With 3D Cloze Test Objective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOL-AE, a Transformer auto-encoder for 3D molecular representation learning that uses positional encodings as atom identifiers in the decoder and a 3D Cloze Test objective (atom dropping plus reconstruction) to replace coordinate denoising.
Problem
Encoder-only 3D molecular pre-training with coordinate denoising suffers from inconsistency between pre-training and downstream objectives and from unstable training because coordinates serve both as content and as atom identifiers.
Approach
Use an auto-encoder whose decoder receives sinusoidal positional encodings as stable atom identifiers, and pre-train by dropping atoms (rather than adding coordinate noise) and reconstructing the 3D structure of the full molecule (3D Cloze Test); discard the decoder for fine-tuning.
Key results
- MOL-AE achieves mean ROC-AUC 79.04 across 9 MoleculeNet classification datasets vs 75.98 for Uni-Mol, with best results on 6 of 9.
- On 6 regression datasets MOL-AE is best on 5 of 6, e.g. FreeSolv RMSE 1.448 vs 1.879 for Uni-Mol and QM7 MAE 53.8 vs 58.9.
- Replacing dropping with coordinate noise (0.5-5 Å) lowers performance; e.g. Tox21 drops from 80.0 to at most 79.5 and HIV from 80.6 to at most 79.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MOL-AE instead of Uni-Mol | 3D molecular property prediction · BACE, BBBP, Tox21, SIDER, HIV, MUV, PCBA, ClinTox, ToxCast scaffold splits | ROC-AUC: 84.1 vs 83.2; 72.0 vs 71.5; 80.0 vs 78.9; 67.0 vs 57.7; 80.6 vs 78.6; 81.6 vs 72.6; 88.9 vs 88.1; 87.8 vs 84.1; 69.6 vs | Helped |
| use MOL-AE instead of Uni-Mol | 3D molecular regression · QM9, QM7, ESOL, FreeSolv, Lipo scaffold splits | MAE or RMSE: 0.0054 -> 0.0053; 58.9 -> 53.8; 0.844 -> 0.830; 1.879 -> 1.448; 0.610 -> 0.607 | Helped |
| use MOL-AE instead of Uni-Mol | 3D molecular property prediction · QM8 scaffold split | MAE: 0.0160 -> 0.0161 | Hurt |
| replace atom dropping with coordinate noise at 0.5 Å (ablation) | MOL-AE · Tox21 | ROC-AUC: 80.0 -> 78.6 | Hurt |
| replace atom dropping with coordinate noise at 1 Å (ablation) | MOL-AE · Tox21 | ROC-AUC: 80.0 -> 79.5 | Hurt |
| replace atom dropping with coordinate noise at 3 Å (ablation) | MOL-AE · Tox21 | ROC-AUC: 80.0 -> 78.9 | Hurt |
| replace atom dropping with coordinate noise at 5 Å (ablation) | MOL-AE · Tox21 | ROC-AUC: 80.0 -> 78.8 | Hurt |
| set decoder depth Ldec to 0 (ablation) | MOL-AE · Tox21 | ROC-AUC: 78.9 -> 74.2 | Hurt |
| set decoder depth Ldec to 1 (ablation) | MOL-AE · Tox21 | ROC-AUC: 78.9 -> 77.7 | Hurt |
| set decoder depth Ldec to 2 (ablation) | MOL-AE · Tox21 | ROC-AUC: 78.9 -> 78.7 | Hurt |
| set decoder depth Ldec to 3 (ablation) | MOL-AE · Tox21 | ROC-AUC: 78.9 -> 77.9 | Hurt |
| add positional encoding to the encoder as well as the decoder (ablation) | MOL-AE with SMILES-order PE · Tox21 | ROC-AUC: 78.9 -> 78.2 | Hurt |
Surprising findings
- Adding positional encoding to the encoder input harms downstream performance, so PE is added only to the decoder.
- Performance does not decrease even at a 60% atom drop ratio.
Limitations
- Evaluated only on MoleculeNet-style property prediction benchmarks.
- Positional encoding derived from SMILES atom order injects order information that may not match 3D structure; random PE performs worse.
Experimental setup
Datasets: Uni-Mol pre-training dataset (19M molecules, 209M conformations) MoleculeNet (BACE, BBBP, Tox21, SIDER, HIV, MUV, PCBA, ClinTox, ToxCast, QM9, QM8, QM7, ESOL, FreeSolv, Lipo)
Benchmarks: MoleculeNet
Metrics: ROC-AUC MAE RMSE
Baselines: D-MPNN Attentive FP N-gram PretrainGNN GROVER GraphMVP MolCLR MoleBLEND Uni-Mol
Builds on: Uni-Mol Transformer-M masked autoencoders (MAE) BERT
Closest papers by meaning
- 3D Denoisers Are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation (AAAI 2025 · similarity 0.88)
- Towards 3D Molecule-Text Interpretation in Language Models (ICLR 2024 · similarity 0.87)
- 3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling (ICLR 2025 · similarity 0.87)
- Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling (ICML 2025 · similarity 0.87)
- Manipulating 3D Molecules in a Fixed-Dimensional E(3)-Equivariant Latent Space (NeurIPS 2025 · similarity 0.87)
- MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.