Research map / Large language models / Agents, code and math
Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChemCoTBench, a step-by-step chemical reasoning benchmark built on modular chemical operations (add/delete/substitute) plus a 22K expert-annotated chain-of-thought dataset, ChemCoTDataset, and evaluates reasoning and non-reasoning LLMs across 22 tasks.
Problem
Existing chemistry benchmarks emphasize factual recall and QA rather than step-by-step reasoning on tasks like molecular optimization and reaction prediction, and they conflate reasoning, knowledge, and calculation.
Approach
Frame chemical problem solving as sequences of modular, verifiable chemical operations on SMILES (addition, deletion, substitution), evaluated across molecule understanding, editing, optimization, and reaction prediction, with a large CoT training set for SFT.
Key results
- Deepseek-R1 shows over 30% improvement over Deepseek-V3 on molecule optimization success rates, per the authors' summary of Table 2.
- Gemini-2.5-pro-think leads foundational tasks, e.g., 0.11 FG MAE and 87.5 Ring-system accuracy.
- SFT on ChemCoTDataset improves Qwen2.5-32B-CoT-SFT Add accuracy from 0.45 to 0.55 and Substitute from 0.5 to 0.53, while prompt-only CoT templates give unstable gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5 models on ChemCoTDataset (ablation) | Qwen2.5 models · Molecule understanding and editing | task scores | Helped |
| Use CoT template prompting (ablation) | Qwen2.5 models · Functional-group detection | MAE: 1.32 -> 0.40; 0.43 -> 0.25; 0.42 -> 0.35; 0.35 -> 0.33 | Helped |
| Use CoT template prompting (ablation) | Qwen2.5 models · Chemical reasoning tasks beyond functional-group detection | task scores | Mixed |
| Enable reasoning mode (ablation) | Qwen3 and Llama-Nemotron models · ChemCoTBench chemical tasks | task performance | No clear effect |
| Distill CoT from DeepSeek-R1 (ablation) | Qwen2.5-Instruct models · ChemCoTBench chemical subtasks | task performance | No clear effect |
| Use reasoning-enhanced rather than non-reasoning models | DeepSeek-R1 versus DeepSeek-V3 · Molecule optimization | performance: >30% | Helped |
| Use reasoning-enhanced rather than non-reasoning models | o3-mini versus GPT-4o · Molecule optimization | performance: >20% | Helped |
Surprising findings
- Enabling thinking mode in open-source hybrid models (Qwen3, Llama-Nemotron) yields no significant gain over non-reasoning counterparts on chemistry tasks.
- Distilled reasoning models (R1-distill, S1.1-distill) and smaller distilled models often perform comparably or worse than their Qwen2.5 backbones on chemistry, with an inverse correlation to math/code performance.
Limitations
- Benchmark relies on a limited set of molecules and tasks (1,495 samples across 22 subtasks).
- CoT annotations depend on DeepSeek-R1 distillation with GPT-4o verification and limited expert review.
- Broader impacts section states no discussion of societal impact.
Experimental setup
Datasets: PubChem ChEMBL ZINC Deep-Mol-Opt USPTO Pistachio Reaxys
Benchmarks: ChemCoTBench
Metrics: MAE Tanimoto similarity accuracy Pass@1 Top-1 accuracy FTS property improvement (Delta) success rate
Baselines: DeepSeek-R1 o1-mini o3-mini Gemini-2.5-pro Claude-3.7-sonnet-thinking Qwen3-235B-A22B GPT-4o DeepSeek-V3 Ether0 BioMedGPT-7B BioMistral-7B Text+Chem T5
Builds on: chain-of-thought prompting Deep-Mol-Opt RDKit
Closest papers by meaning
- Structured Chemistry Reasoning with Large Language Models (ICML 2024 · similarity 0.91)
- Training a Scientific Reasoning Model for Chemistry (NeurIPS 2025 · similarity 0.91)
- Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation (NeurIPS 2024 · similarity 0.91)
- LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning (ICML 2026 · similarity 0.91)
- From What to Why: A Multi-Agent System for Evidence-based Chemical Reaction Condition Reasoning (ICLR 2026 · similarity 0.89)
- LLM-Augmented Chemical Synthesis and Design Decision Programs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.