Research map / Large language models / Agents, code and math
CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics
Weida Wang, Dongchen Huang, Jiatong LI, Tengchao Yang, Ziyang Zheng, Chuyi Peng, Di Zhang, Dong Han, Benteng Chen, Binzhao Luo, Zhiyu Liu, kunling liu and 24 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SEED instead of OlympiadBench-rule metric for agreement with human experts | evaluation metric · Spearman correlation with human ratings | Spearman correlation: 0.41 -> 0.90 | Improved |
| use SEED instead of GPT-4o judgment | evaluation metric · Spearman correlation with human ratings | Spearman correlation: 0.56 -> 0.90 | Improved |
| use SEED partial credit instead of strict expert-labeled accuracy | model scoring · CMPhysBench | score: 28.9 -> 36.0 (Grok 4) | Improved |
| use reasoning (long-CoT) models instead of general LLMs | LLMs · CMPhysBench | SEED score | Mixed |
Closest papers by meaning
- Fine-Tuned Language Models Generate Stable Inorganic Materials as Text (ICLR 2024 · similarity 0.85)
- Scaling Physical Reasoning with the PHYSICS Dataset (NeurIPS 2025 · similarity 0.84)
- PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems? (NeurIPS 2026 · similarity 0.84)
- Diffusion-warm sampling of the XY model enables fast thermalization at scale (NeurIPS 2026 · similarity 0.84)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.84)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
