Research map / Large language models / AI and society
Assessing Large Language Models on Climate Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a science-communication-grounded framework and human rating study with AI-assisted, education-screened raters to evaluate LLM answers to 300 climate questions on 8 presentational and epistemological dimensions.
Problem
Existing evaluations do not assess how well LLMs answer climate-change questions in terms of both presentational comprehensibility and epistemological adequacy (accuracy, specificity, completeness, uncertainty), and human raters struggle to detect issues without support.
Approach
A science-communication-based rating framework (4 presentational + 4 epistemological dimensions) combined with a scalable-oversight protocol in which raters receive GPT-4-generated critiques grounded in verbatim Wikipedia evidence (AI Assistance); the study also compares with attribution (AIS) evaluation.
Key results
- Presentational quality is high for most LLMs (e.g., ChatGPT style 4.54, clarity 4.56, correctness 4.58 on a 1-5 scale) but tone is markedly lower (about 3.0-3.4 for most models).
- Epistemological scores are consistently lower; for example, completeness ranges 2.26-2.73 and uncertainty 2.05-2.55 across models, with Falcon-180B-Chat best on accuracy (3.81) and completeness (2.73).
- Human raters detected 77% of planted issues with AI Assistance versus 60% without; AIS attribution showed near-zero Spearman correlation with epistemological ratings (|rho| <= 0.06).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | tone: 3.34 -> 3.59 | Helped |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | accuracy: 3.77 -> 3.92 | Helped |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | specificity: 3.29 -> 3.46 | Helped |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | completeness: 2.72 -> 2.97 | Helped |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | uncertainty: 2.73 -> 2.92 | Helped |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | style: 4.33 -> 4.10 | Hurt |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | clarity: 4.34 -> 4.24 | Hurt |
| Add evaluation criteria to the prompt (ablation) | GPT-4 · Climate questions | correctness: 4.43 -> 4.29 | Hurt |
| Show raters AI Assistance | Human raters evaluating climate answers · Manually constructed examples with issues | issues detected: 60% -> 77% | Helped |
| Provide previous exposure to AI Assistance but no assistance in the current rating (ablation) | Human raters evaluating GPT-4 answers · Climate questions | accuracy rating: 4.49 -> 3.86 | Hurt |
Surprising findings
- Epistemological and presentational quality show a trade-off: GPT-4 with dimension-aware prompts improved epistemological scores but lowered most presentational scores.
- AIS attribution is largely orthogonal to epistemological quality; fully attributed answers can still score low on epistemological dimensions.
Limitations
- Human rating is subjective with only medium inter-rater agreement and no gold ratings for calibration.
- AI Assistance may bias raters and may introduce false issues; GPT-4 is used both for assistance and as an evaluated model.
- Answers are limited to short 3-4 sentence responses, which may penalize comprehensiveness.
Experimental setup
Datasets: Google Trends climate questions Skeptical Science myths Wikipedia climate articles (synthetic GPT-4 questions)
Benchmarks: 300 climate change questions (Google Trends, Skeptical Science, Wikipedia-synthetic)
Metrics: mean rating per dimension bootstrapped 95% confidence intervals pairwise t-tests issue detection rate Krippendorff's alpha Spearman correlation (AIS vs epistemological ratings)
Baselines: ChatGPT InstructGPT (text-davinci-003) InstructGPT (text-davinci-002) InstructGPT (turbo) PaLM-2 (text-bison) Falcon-180B-Chat
Builds on: Attribution to Identified Source (AIS) Multidimensional Quality Metrics (MQM) scalable oversight in-context probing
Closest papers by meaning
- ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models (ICLR 2025 · similarity 0.90)
- AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science (NeurIPS 2025 · similarity 0.89)
- Adapting While Learning: Grounding LLMs for Scientific Problems with Tool Usage Adaptation (ICML 2025 · similarity 0.85)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.85)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.85)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.