Research map / Large language models / Agents, code and math
Structured Chemistry Reasoning with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StructChem, a three-phase prompting strategy (formulae generation, step-by-step program-aided reasoning, confidence-based review-and-refinement) that substantially boosts GPT-4 and GPT-3.5 on complex SciBench chemistry problems, and fine-tuned Llama-2-13B/Vicuna-13B on its outputs.
Problem
LLMs struggle with complex, multi-step chemistry problems requiring domain formulae, precise calculation, and compositional reasoning, and their errors often stem from missing reasoning structure rather than missing knowledge.
Approach
Decompose chemistry reasoning into formulae recall, grounded step-by-step reasoning (with PoT Python code), and a confidence-scored iterative review-and-refine loop that only accepts higher-confidence revisions, so correct answers are not swayed by unfaithful feedback.
Key results
- StructChem with GPT-4 reaches 30.11 average zero-shot and 47.64 few-shot accuracy, versus 26.69 and 32.22 for the best baselines (CoT and PoT).
- Fine-tuning Vicuna-13B and Llama-2-13B on StructChem outputs yields averages of 22.52 and 21.61, versus 10.42 and 11.07 for CoT-generated data.
- Manual error analysis finds irrelevant formulae (25.9%) more common than incorrect formulae (13.7%), and reasoning errors still account for about 35% of errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use StructChem instead of the strongest listed baseline | GPT-4 · quan, zero-shot | accuracy: 8.82 -> 20.59 | Helped |
| Use StructChem instead of the strongest listed baseline | GPT-4 · chemmc, zero-shot | accuracy: 43.59 -> 38.46 | Hurt |
| Use StructChem instead of the strongest listed baseline | GPT-4 · atkins, zero-shot | accuracy: 28.04 -> 31.78 | Helped |
| Use StructChem instead of the strongest listed baseline | GPT-4 · matter, zero-shot | accuracy: 22.45 -> 24.49 | Helped |
| Use StructChem instead of the strongest listed baseline | GPT-4 · quan, few-shot | accuracy: 38.24 -> 41.18 | Helped |
| Use StructChem instead of the strongest listed baseline | GPT-4 · chemmc, few-shot | accuracy: 46.15 -> 58.97 | Helped |
| Use StructChem instead of the strongest listed baseline | GPT-4 · atkins, few-shot | accuracy: 21.05 -> 59.81 | Helped |
| Use StructChem instead of the strongest listed baseline | GPT-4 · matter, few-shot | accuracy: 28.57 -> 30.67 | Helped |
| Add formulae review to structured instruction (ablation) | StructChem · SciBench four-dataset average, zero-shot | accuracy: 29.26 -> 30.87 | Helped |
| Add formulae review to structured instruction (ablation) | StructChem · SciBench four-dataset average, few-shot | accuracy: 41.37 -> 41.56 | Helped |
| Add reasoning review to structured instruction (ablation) | StructChem · SciBench four-dataset average, zero-shot | accuracy: 29.26 -> 32.03 | Helped |
| Add confidence scoring to review (ablation) | StructChem · SciBench four-dataset average, few-shot | accuracy: 42.93 -> 45.20 | Helped |
Surprising findings
- Errors often stem from missing reasoning structure rather than missing domain knowledge.
- Irrelevant formulae are more frequent than incorrect formulae, and relevance may matter more than correctness for final accuracy.
Limitations
- Evaluation limited to four small SciBench subsets with few test problems (e.g., 8 to 16 solved problems per set in the statistics table).
- Reasoning errors and calculation errors remain common even with the method.
- Relies on the LLM's own parametric chemistry knowledge without external retrieval.
Experimental setup
Datasets: SciBench quan SciBench chemmc SciBench atkins SciBench matter GPT-4-generated chemistry problems (1,000)
Benchmarks: SciBench quan SciBench chemmc SciBench atkins SciBench matter
Metrics: accuracy error proportion
Baselines: Direct reasoning System instruction CoT PoT
Builds on: chain-of-thought program-of-thoughts self-consistency LoRA SciBench
Closest papers by meaning
- Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations (NeurIPS 2025 · similarity 0.91)
- Training a Scientific Reasoning Model for Chemistry (NeurIPS 2025 · similarity 0.90)
- Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation (NeurIPS 2024 · similarity 0.90)
- LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning (ICML 2026 · similarity 0.89)
- ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area (AAAI 2025 · similarity 0.87)
- MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.