Research map / Large language models / Agents, code and math
FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FGBench, a 625K-QA dataset built from MoleculeNet molecular pairs with functional-group-level annotations, and benchmarks nine LLMs on a 7,146-question subset, finding that current LLMs struggle with FG-level property reasoning.
Problem
Existing chemistry LLM datasets center on molecule-level property labels and lack explicit links between functional groups and property changes, limiting FG-level property reasoning in LLMs.
Approach
Builds QA pairs from molecular pairs (similarity >0.7) whose functional-group differences are annotated with AccFG and validated by a validation-by-reconstruction step, covering single-FG impact, multi-FG interaction, and whole-molecule comparison tasks in Boolean and value formats.
Key results
- o3-mini reaches 0.687 accuracy on single-FG Boolean questions, while GPT-4o drops from 0.667 (Single) to 0.488 (Interaction) accuracy.
- Llama-3.1 70B falls from 0.683 to 0.530 accuracy between the Single and Interaction Boolean categories.
- nach0-base predicted 'False' for 97.7% of single-FG Boolean samples, so its 0.606 accuracy is largely a bias artifact.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ask about multiple functional group interactions rather than a single functional group impact | LLMs evaluated on FGBench · FGBench Boolean tasks | accuracy: 0.667 to 0.488 | Hurt |
| Ask about multiple functional group interactions rather than a single functional group impact | Llama-3.1 70B · FGBench Boolean tasks | accuracy: 0.683 to 0.53 | Hurt |
| Use a larger model (70B rather than 8B) | Llama-3.1 · FGBench | accuracy | Mixed |
| Fine-tune for chemistry tasks rather than use a general model | ChemLLM-7B compared with Llama-3.1 8B · FGBench | accuracy | Hurt |
| Provide molecular comparison questions rather than functional-group interaction questions | Qwen2.5-7B · FGBench Boolean tasks | accuracy: 0.396 to 0.664 | Helped |
Surprising findings
- Chemistry-specialized models (ChemLLM-7B, nach0, Llama-3-8B-MolInst, LlaSMol) underperform general-purpose LLMs on FG-level reasoning despite training on molecule data.
- Some models (o3-mini, Qwen2.5-7B) perform better on Comparison than on FG-level tasks, possibly because MoleculeNet was seen in pretraining.
Limitations
- Does not explicitly model position isomerism, carbon chain isomerism, or stereoisomerism.
- Benchmark uses a 7,146-question subset of the 625K dataset and evaluates only nine models.
- Functional group definitions are context-dependent and follow AccFG's descriptor vocabulary.
Experimental setup
Datasets: ESOL Lipophilicity FreeSolv HIV BACE BBBP Tox21 SIDER ClinTox QM9
Benchmarks: FGBench
Metrics: accuracy RMSE validity rate
Baselines: ChemLLM-7B nach0 Llama-3-8B-MolInst LlaSMol-Mistral-7B Qwen2.5-7B
Builds on: AccFG MoleculeNet Mol-Instructions SMolInstruct
Closest papers by meaning
- Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations (NeurIPS 2025 · similarity 0.88)
- Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation (NeurIPS 2024 · similarity 0.88)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.88)
- TRIDENT: Tri-Modal Molecular Representation Learning with Taxonomic Annotations and Local Correspondence (NeurIPS 2025 · similarity 0.87)
- KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledge (NeurIPS 2025 · similarity 0.87)
- Hierarchical Graph Tokenization for Molecule-Language Alignment (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.