Research map / Large language models / Agents, code and math
Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChemCost, a judge-free benchmark of 1,427 reactions for LLM agents to estimate chemical procurement cost via tools, with noise-injected views and stage-level diagnostics.
Problem
Evaluating scientific tool-using LLM agents on chemical procurement cost estimation lacks exact, judge-free ground truth; agents must ground chemicals, retrieve supplier quotes, select packs, and compute costs from reaction text.
Approach
Deterministic cost labels computed from a frozen 230,775-quote supplier snapshot under a fixed pack-selection rule, paired with four-stage noise injection (names, quantities, missing fields, format) and stage-level trajectory diagnostics.
Key results
- Qwen3.5-Plus achieves the highest clean CTA@25 at 50.6%, while DeepSeek V4 Pro leads CTA@10 at 37.0%.
- Format noise (+Fmt) raises abstention by 26.3 percentage points on average across frontier agents, more than any other noise stage.
- Tool-free methods (ZeroShot, FewShot, CoT) reach at most 7.4% CTA@25, versus 34.6-50.6% with ReAct tools.
Surprising findings
- Chemistry-specialized models (ChemDFM-v2.0-14B, ChemLLM-20B) achieve high component precision and recall but near-zero CTA scores.
- Tool-call count correlates strongly with accuracy across models (r=0.98), yet failed trajectories often use more calls than successful ones.
Limitations
- Labels abstract away stock availability, shipping, regional pricing, lead time, and bulk discounts.
- Evaluation limited to ReAct-style agents with a fixed tool set and 40-step budget.
- Human reference based on only three chemists and about 30 hours of annotation.
Experimental setup
Datasets: Open Reaction Database organic chemistry textbooks Organic Syntheses PaRoutes ChemPU hand-crafted examples MolPort ChemSpace Sigma-Aldrich
Benchmarks: ChemCost
Metrics: CTA@10 CTA@25 component precision component recall abstention rate stage-level pass-through rate
Baselines: ZeroShot FewShot Chain-of-Thought ChemDFM-v2.0-14B ChemLLM-20B LlaSMol-7B
Builds on: ReAct ChemCrow Coscientist
Closest papers by meaning
- From What to Why: A Multi-Agent System for Evidence-based Chemical Reaction Condition Reasoning (ICLR 2026 · similarity 0.90)
- Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations (NeurIPS 2025 · similarity 0.88)
- LLM-Augmented Chemical Synthesis and Design Decision Programs (ICML 2025 · similarity 0.86)
- MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses (ICLR 2025 · similarity 0.86)
- LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning (ICML 2026 · similarity 0.86)
- Training a Scientific Reasoning Model for Chemistry (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.