Research map / Large language models / Agents, code and math
Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MolPuzzle, a multimodal benchmark of 217 molecular structure elucidation puzzles (spectra plus formula) organized into three sequential stages, and evaluates 12 LLMs and a human baseline.
Problem
LLMs' multi-step chemical reasoning is underexplored; molecular structure elucidation from spectral data (IR, MS, 1H/13C NMR) requires integrating multiple clues and iterative hypothesis testing.
Approach
Frames molecular structure elucidation as a crossword-like puzzle with three interlinked stages (molecule understanding, spectrum interpretation, molecule construction), with template-based QA generated via RDKit and evaluated zero-shot against a human expert baseline.
Key results
- GPT-4o achieves near-100% accuracy on Stage-1 molecule understanding (SI F1 1.00) but only 1.4% exact-match accuracy on full puzzle construction, versus 66.7% for human experts.
- On Stage-2 spectrum interpretation, GPT-4o reaches F1 around 0.61-0.66 across IR, MASS, H-NMR, and C-NMR, compared with human baselines of roughly 0.73-0.77.
- GPT-4o's MACCS fingerprint Tanimoto similarity of 0.431 on full puzzles indicates chemically related but mostly incorrect molecules; all generated molecules were valid.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of the other evaluated LLMs | LLMs solving complete MolPuzzle instances · MolPuzzle entire puzzles | Acc. (exact match): 0.013±0.008 (Claude-3-opus) -> 0.014±0.004 (GPT-4o) | Helped |
| Use GPT-4o instead of human experts | Solving complete MolPuzzle instances · MolPuzzle entire puzzles | Acc. (exact match): 0.667±0.447 -> 0.014±0.004 | Hurt |
| Use GPT-4o for spectrum interpretation rather than human experts | MolPuzzle spectrum interpretation · MolPuzzle spectrum interpretation: IR | F1: 0.753±0.221 (Human) -> 0.656±0.052 (GPT-4o) | Hurt |
| Use GPT-4o for molecule construction rather than human experts | MolPuzzle molecule construction · MolPuzzle molecule construction: H-NMR elucidation | F1: 0.867±0.230 (Human) -> 0.524±0.021 (GPT-4o) | Hurt |
| Add chain-of-thought prompting (ablation) | GPT-4o on MolPuzzle · MolPuzzle molecule understanding: FGI | F1: 0.934±0.005 -> 0.689±0.025 | Hurt |
| Add chain-of-thought prompting (ablation) | GPT-4o on MolPuzzle · MolPuzzle molecule understanding: SDC | F1: 0.667±0.003 -> 0.816±0.062 | Helped |
| Add chain-of-thought prompting (ablation) | GPT-4o on MolPuzzle · MolPuzzle molecule understanding: ARI | F1: 0.943±0.016 -> 0.911±0.031 | Hurt |
| Add chain-of-thought prompting (ablation) | GPT-4o on MolPuzzle · MolPuzzle molecule understanding: SI | F1: 1±0.0 -> 1±0.0 | No clear effect |
| Add chain-of-thought prompting (ablation) | GPT-4o on MolPuzzle · MolPuzzle entire puzzles | Acc. (exact match) | Hurt |
Surprising findings
- Chain-of-thought prompting degraded performance on spectrum interpretation and molecule construction compared with zero-shot prompting.
- Although GPT-4o nearly solves molecule understanding, it collapses on spectrum interpretation and full molecule construction.
Limitations
- Exact-match accuracy is strict; fragment pools were partly LLM-generated and only 67.4% were judged correct by experts before manual correction.
- Human baseline is small (six graduate students), with high variance on some tasks.
- NMR spectra in Stage 3 were provided as verified simulated data from nmrdb rather than real spectra, and some MS spectra were simulated.
Experimental setup
Datasets: Organic Structures from Spectra (textbook molecules) PubChem spectra
Benchmarks: MolPuzzle
Metrics: F1 accuracy exact match accuracy Levenshtein distance validity MACCS FTS RDK FTS Morgan FTS
Baselines: GPT-3.5-turbo Claude-3-opus Gemini-pro Llama-3-8B-Instruct LLaVA-Llama-3-8B Qwen-VL-Chat InstructBLIP Galactica-30b Vicuna-13B-v1.5 Mistral-7B-Instruct-v0.3
Builds on: RDKit chain-of-thought
Closest papers by meaning
- Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations (NeurIPS 2025 · similarity 0.91)
- Structured Chemistry Reasoning with Large Language Models (ICML 2024 · similarity 0.90)
- Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model (NeurIPS 2025 · similarity 0.89)
- MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses (ICLR 2025 · similarity 0.89)
- FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models (NeurIPS 2025 · similarity 0.88)
- ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.