Research map / Large language models / Agents, code and math
AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AtmosSci-Bench, a template-based MCQ plus open-ended QA benchmark for LLMs in atmospheric and related Earth sciences, and evaluates 20+ instruction, reasoning, math, and climate-domain models.
Problem
Existing LLM benchmarks (GSM8K, MATH, GPQA, etc.) do not assess the numerical, symbolic, and interdisciplinary reasoning needed for atmospheric science, and there is no comprehensive benchmark for this domain.
Approach
Symbolic template-based MCQ generation (expanding 67 expert-written templates into 10 or 30 instances with rule-based numeric constraints and automatic solvers) combined with open-ended questions graded by a cascade of quantity, symbolic, and LLM-as-judge evaluators.
Key results
- GPT-o3-mini reaches 89.1% overall accuracy on MCQ10, the best reasoning model; the top instruction model Gemini-2.0-Flash-Exp reaches 64.18%.
- ClimateGPT-7B and ClimateGPT-70B score 20.15% and 27.31% on MCQ10, below general instruction models; GeoGPT-Qwen2.5-72B scores 34.93% versus 57.01% for its Qwen2.5-72B base.
- On OEQ, all models score below 40%, with the best (GPT-o3-mini) at 33.76%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reasoning-optimized models rather than instruction models | LLMs evaluated on AtmosSci-Bench MCQ10 · MCQ10 | accuracy: 70.9% - 89.1% vs 18.5% - 64.18%; 24.92% gap | Helped |
| Use reasoning-optimized models rather than math models | LLMs evaluated on AtmosSci-Bench MCQ10 · MCQ10 | accuracy | Helped |
| Use reasoning-optimized models rather than domain-specific climate models | LLMs evaluated on AtmosSci-Bench MCQ10 · MCQ10 | accuracy | Helped |
| Increase reasoning token limits from 4K toward 16K | QwQ-32B-Preview · MCQ10 | accuracy | Helped |
| Increase reasoning token limits beyond 16K | QwQ-32B-Preview · MCQ10 | accuracy | No clear effect |
| Increase reasoning token limits beyond 32K | GPT-o3-mini · MCQ10 | accuracy | No clear effect |
| Increase symbolic variation of question values | Reasoning models evaluated on MCQ30 · MCQ30 | accuracy | Hurt |
| Use reasoning-aligned models rather than instruction-only counterparts | LLMs evaluated on OEQ · OEQ | accuracy | Helped |
| Use GeoGPT-Qwen2.5-72B rather than Qwen2.5-72B-Instruct-Turbo | Qwen2.5-72B-Instruct-Turbo · MCQ10 | accuracy: 57.01% -> 34.93% | Hurt |
Surprising findings
- Domain-specific climate models (ClimateGPT, GeoGPT) underperform general instruction models of similar size.
- Accuracy on the original questions falls about one standard deviation (QwQ about two) from the mean over symbolic perturbations, a result the authors read as possible contamination or pattern matching.
Limitations
- OEQ set is small (391 questions) and drawn from one university's course materials, limiting coverage.
- Reasoning models were run with a 32K context cap and default decoding parameters, and some results rely on preview or experimental API endpoints.
- The LLM-as-judge evaluator (GPT-4o-mini) may introduce grading errors, and sympy-based checks can produce false negatives.
Experimental setup
Datasets: university atmospheric science course materials
Benchmarks: AtmosSci-Bench MCQ10 AtmosSci-Bench MCQ10_EXT AtmosSci-Bench OEQ AtmosSci-Bench MCQ30
Metrics: accuracy symbolic standard deviation
Baselines: GPT-4o GPT-o1 GPT-o3-mini DeepSeek-R1 QwQ-32B-Preview Gemini-2.0-Flash-Thinking-Exp ClimateGPT GeoGPT
Builds on: GSM-Symbolic PHYSICS benchmark
Closest papers by meaning
- Assessing Large Language Models on Climate Information (ICML 2024 · similarity 0.89)
- ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models (ICLR 2025 · similarity 0.87)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.86)
- WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning (ICLR 2025 · similarity 0.85)
- Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models (NeurIPS 2024 · similarity 0.84)
- Adapting While Learning: Grounding LLMs for Scientific Problems with Tool Usage Adaptation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.