Research map / Large language models / Agents, code and math
ChemEval: A Multi-level and Fine-grained Chemical Capability Evaluation for Large Language Models
Yuqing Huang, Rongyang Zhang, Xuesong He, Xuyang Zhi, Hao Wang, Nuo Chen, Zongbo Liu, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, YiLi and 7 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3-shot examples to the prompt (ablation) | OpenAI-o1 · ReactTR | F1: 25.00 -> 60.00 | Improved |
| add 3-shot examples to the prompt (ablation) | GPT-4o · ReactTR | F1: 32.00 -> 71.00 | Improved |
| add 3-shot examples to the prompt (ablation) | DeepSeek-V3 · SubE | Accuracy: 61.26 -> 75.78 | Improved |
| add 3-shot examples to the prompt (ablation) | Llama3.3-8B · SATask | LLM Score: 38.40 -> 29.00 | Hurt |
| add 3-shot examples to the prompt (ablation) | ChemDFM · TempE | F1: 74.33 -> 41.00 | Hurt |
| add 3-shot examples to the prompt (ablation) | ChemSpark · LRec | F1: 37.60 -> 16.80 | Hurt |
| scale Qwen2.5 from 7B to 72B (ablation) | Qwen2.5 · MCTask | Accuracy: 59.60 -> 67.20 | Improved |
| scale Qwen2.5 from 7B to 72B (ablation) | Qwen2.5 · CalcTask | LLM Score: 43.60 -> 61.90 | Improved |
| scale Qwen2.5 from 7B to 72B (ablation) | Qwen2.5 · MolPC | Accuracy: 64.04 -> 48.13 | Hurt |
| scale Qwen2.5 from 7B to 72B (ablation) | Qwen2.5 · CatRec | F1: 0.00 -> 0.00 | No clear effect |
| use thinking model OpenAI-o1 instead of GPT-4o | GPT-4o · MCTask | Accuracy: 66.80 -> 74.00 | Improved |
| use thinking model DeepSeek-R1 instead of DeepSeek-V3 | DeepSeek-V3 · MCTask | Accuracy: 76.00 -> 82.40 | Improved |
Closest papers by meaning
- ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area (AAAI 2025 · similarity 0.90)
- Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations (NeurIPS 2025 · similarity 0.90)
- ChemPile: A 250 GB Diverse and Curated Dataset for Chemical Foundation Models (NeurIPS 2025 · similarity 0.89)
- Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning (NeurIPS 2026 · similarity 0.89)
- LLM-Augmented Chemical Synthesis and Design Decision Programs (ICML 2025 · similarity 0.89)
- Structured Chemistry Reasoning with Large Language Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
