Research map / Vision-language and multimodal / Human and embodied modeling
Towards 3D Molecule-Text Interpretation in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D-MoLM, which equips Llama2-7B with a frozen Uni-Mol 3D molecular encoder linked via a Q-Former projector, trained with three stages on a new 3D-MoIT instruction dataset built from PubChem and PubChemQC.
Problem
Language models cannot interpret 3D molecular structures, which limits their use on 3D-dependent biomolecular properties and tasks such as molecule-text retrieval, captioning, and molecular QA.
Approach
Integrate a pretrained 3D molecular encoder (Uni-Mol) into an LM through a BLIP-2-style Q-Former projector that yields soft molecular tokens, combined with a 3D molecule-centric instruction tuning dataset that includes DFT-computed 3D-dependent properties (HOMO, LUMO, gap, SCF energy).
Key results
- On the PubChem Dataset, 3D-MoLM reaches 72.06% M2T accuracy in the test set for retrieval versus 62.07% for the best 2D baseline MoMu-K (dagger variant).
- On molecule captioning, the 3D-MoLM specialist achieves BLEU-2 30.32 and ROUGE-L 31.23, above MolT5-Large (25.87 and 23.41) and MoMu-Large (26.34 and 24.76).
- On computed property QA, the 3D-MoLM specialist reaches HOMO-LUMO gap MAE 0.28 eV, versus 0.88 eV for Llama2-7B and 0.67 eV for 2D-MoLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3D molecular perception instead of 2D graph perception | 3D-MoLM · PubChem Dataset molecule captioning, specialist | ROUGE-L: 24.76 -> 31.23 | Helped |
| use 3D molecular perception instead of 2D graph perception | 3D-MoLM · 3D-MoIT descriptive property QA, specialist | BLEU-2: 30.84 -> 32.00 | Helped |
| use 3D molecular perception instead of 2D graph perception | 3D-MoLM · 3D-MoIT computed property QA, specialist | H-L Gap MAE (eV): 0.67 -> 0.28 | Helped |
| add 1D SMILES to prompts containing 3D molecular tokens (ablation) | 3D-MoLM · PubChem Dataset molecule captioning | BLEU-2: 27.12 -> 29.25 | Helped |
| add 1D SMILES to prompts containing 3D molecular tokens (ablation) | 3D-MoLM · 3D-MoIT descriptive property QA | BLEU-2: 28.05 -> 31.61 | Helped |
| add 1D SMILES to prompts containing 3D molecular tokens (ablation) | 3D-MoLM · 3D-MoIT computed property QA | H-L Gap MAE (eV): 0.48 -> 0.32 | Helped |
| add a 3D molecular encoder and projector (ablation) | MolT5-small · PubChem Dataset molecule captioning | BLEU-2: 22.53 -> 23.73 | Helped |
| add a 3D molecular encoder and projector (ablation) | MolT5-base · PubChem Dataset molecule captioning | BLEU-2: 24.51 -> 25.55 | Helped |
| add a 3D molecular encoder and projector (ablation) | MolT5-large · PubChem Dataset molecule captioning | BLEU-2: 25.87 -> 26.91 | Helped |
| use GPT-3.5-enriched rather than original PubChem texts for pretraining (ablation) | 3D-MoLM · PubChem Dataset test set molecule-text retrieval | M2T accuracy: 72.06 -> 69.05 | Hurt |
| use GPT-3.5-enriched rather than original PubChem texts for pretraining (ablation) | 3D-MoLM · PubChem Dataset molecule captioning, specialist | BLEU-2: 29.82 -> 30.32 | Helped |
Surprising findings
- GPT-3.5 enrichment of PubChem descriptions lowered retrieval accuracy on the test set (for example, 3D-MoLM M2T 72.06 with the original texts vs 69.05 with enrichment), though it improved captioning and QA.
- Adding 3D conformation perception to MolT5 at each scale improved captioning metrics, which suggests the gain is not due only to the larger LM.
Limitations
- Dataset scale is small (about 300K PubChem pairs and 3M PubChemQC entries) compared with vision-language datasets.
- Fine-grained discrimination of similar molecules and atom counting is weak, as shown in failure cases.
- Conformations for PubChem molecules come from RDKit MMFF rather than DFT ground truth, and in-context and chain-of-thought abilities are not explored.
Experimental setup
Datasets: PubChem PubChemQC PCDes ChEBI-20
Benchmarks: PubChem Dataset 3D-MoIT
Metrics: Accuracy Recall@20 BLEU-2 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L METEOR MAE valid answer rate
Baselines: Sci-BERT KV-PLM MoMu MolT5 Llama2-7B 2D-MoLM Uni-Mol
Builds on: BLIP-2 Uni-Mol LLaMA 2 MoMu InstructBLIP
Closest papers by meaning
- 3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling (ICLR 2025 · similarity 0.94)
- Improving Large Molecular Language Model via Relation-aware Multimodal Collaboration (AAAI 2026 · similarity 0.88)
- Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models (ICLR 2024 · similarity 0.88)
- Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model (NeurIPS 2025 · similarity 0.88)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.88)
- NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.