Research map / Large language models / Agents, code and math
URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces URSA, a route-level benchmark that scores stock-terminated retrosynthetic routes for chemical plausibility (Solv-0/1/2) using ChemCensor, and benchmarks conventional CASP tools and frontier/open LLMs on expert and drug target sets.
Problem
Standard retrosynthesis evaluation relies on solvability (reaching purchasable starting materials), which does not ensure that each reaction step is chemically plausible; existing benchmarks such as USPTO-190 are also weakly representative of prospective medicinal chemistry.
Approach
A route-level validity protocol that combines stock termination, structural/topological checks, and step-wise ChemCensor precedent-based plausibility scoring (Solv-2), applied uniformly to heterogeneous CASP tools and LLM planners, with a new expert-derived reaction plausibility benchmark used to justify ChemCensor over LLM judges.
Key results
- ChemCensor reaches 0.96 accuracy and 0.92 MCC on the 1000-reaction expert plausibility benchmark, versus 0.83 accuracy (0.67 MCC) for the best LLM judge, Gemini 3.1 Pro.
- On URSA-expert-2026 the best Solv-2 scores are RetroChimera-MCTS 32%, AZF-MCTS 29%, and GPT 5.5 at 21% among LLMs.
- On URSA-drugs&clinicals-2026, RetroChimera-MCTS reaches 60% Solv-2, while DreamRetroer reaches 95%+ Solv-0 on expert set but only 9% Solv-2.
Surprising findings
- LLM judges show a systematic bias toward labeling reactions plausible, with non-plausible recall of only 0.50-0.73.
- Kimi K2.5 reaches 97% Solv-0 on the expert set but only 5% Solv-2, showing stock-termination can overstate validity.
Limitations
- Evaluates up to Solv-2 only; Solv-3 executability (conditions, yields, safety) is not scored.
- ChemCensor precedent database is primarily patent-derived (USPTO), so judgments inherit its coverage and biases.
- Metrics computed only on the best route per target, not full route diversity or failure distributions.
Experimental setup
Datasets: USPTO-full Pistachio 2023Q4 Enamine building blocks ASKCOS buyables eMolecules building blocks URSA-expert-2026 URSA-drugs&clinicals-2026 USPTO-190
Benchmarks: URSA-expert-2026 URSA-drugs&clinicals-2026 USPTO-190 URSA-reaction-plausibility-bench-2026
Metrics: accuracy MCC Solv-0 Solv-1 Solv-2 CC*
Baselines: Retro* DirectMultiStep ASKCOS AiZynthFinder RetroChimera SynPlanner DreamRetroer TTLA SynLlama Gemini 3.1 Pro GPT 5.5 Claude Opus 4.8
Builds on: ChemCensor RetroCast Syntheseus PaRoutes USPTO-190
Closest papers by meaning
- LLM-Augmented Chemical Synthesis and Design Decision Programs (ICML 2025 · similarity 0.90)
- Trustworthy Retrosynthesis: Eliminating Hallucinations with a Diverse Ensemble of Reaction Scorers (NeurIPS 2026 · similarity 0.89)
- Double-Ended Synthesis Planning with Goal-Constrained Bidirectional Search (NeurIPS 2024 · similarity 0.87)
- RetroOOD: Understanding Out-of-Distribution Generalization in Retrosynthesis Prediction (AAAI 2024 · similarity 0.87)
- Retrieval-Retro: Retrieval-based Inorganic Retrosynthesis with Expert Knowledge (NeurIPS 2024 · similarity 0.86)
- RETRO SYNFLOW: Discrete Flow-Matching for Accurate and Diverse Single-Step Retrosynthesis (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.