Research map / Large language models / Agents, code and math
CME–SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
Jin Gene Wong, Anjney Midha, Joseph Tennyson, Wei-Lin Chiang, Ion Stoica, Zhi-Xun Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CME–SpectrumBench, an expert-designed benchmark of 190 condensed matter spectral data analysis questions for evaluating LLMs on experimental data analysis.
Problem
Existing LLM benchmarks emphasize theoretical, symbolic reasoning in math and physics, while experimental data analysis in frontier research (here, condensed matter spectroscopy) is largely unevaluated.
Approach
An expert-designed benchmark of 190 questions focused on analyzing condensed matter spectral data, targeting data analysis rather than symbolic theoretical reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| give the LLM a Python interpreter with or without ReAct prompting (ablation) | best frontier model · CME-SpectrumBench, all spectra | Lorentzian score: 0.34 -> 0.46 | Improved |
| give the LLM a Python interpreter with or without ReAct prompting (ablation) | best frontier model · CME-SpectrumBench, minimal noise/convolution | Lorentzian score: 0.48 -> 0.68 | Improved |
| give the LLM a Python interpreter with or without ReAct prompting (ablation) | frontier models · CME-SpectrumBench, maximum noise/convolution | Lorentzian score | No clear effect |
| give the LLM a Python interpreter with or without ReAct prompting (ablation) | Qwen3 · CME-SpectrumBench | Lorentzian score | No clear effect |
| give the LLM a Python interpreter with or without ReAct prompting (ablation) | Claude Opus and Sonnet 4.5 · CME-SpectrumBench | Lorentzian score | Improved |
| increase dataset resolution (token count) (ablation) | Gemini 3.1 Pro, direct query · CME-SpectrumBench noiseless questions | normalized score | Hurt |
| increase noise level (ablation) | direct-query models · CME-SpectrumBench | Lorentzian score | Hurt |
| add a small amount of noise at low noise levels (ablation) | Gemini 2.5 Pro Preview, question B2 · CME-SpectrumBench | Lorentzian score | Mixed |
| advanced prompting on Tier II array-extraction questions (ablation) | frontier models · CME-SpectrumBench Tier II | Lorentzian score | Improved |
Experimental setup
Benchmarks: CME–SpectrumBench
Closest papers by meaning
- A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules? (ICML 2024 · similarity 0.87)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.87)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.86)
- LLM-SR: Scientific Equation Discovery via Programming with Large Language Models (ICLR 2025 · similarity 0.85)
- Scaling Physical Reasoning with the PHYSICS Dataset (NeurIPS 2025 · similarity 0.84)
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
