Research map / Large language models / Retrieval and knowledge
ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ClimaGen, a semi-automated framework that generates climate QA from graduate textbooks with expert validation, releasing the ClimaQA-Gold (566 expert-validated) and ClimaQA-Silver (3000 synthetic) benchmarks and evaluating LLMs with RAG, fine-tuning and continued pre-training.
Problem
Existing scientific QA benchmarks either depend heavily on manual expert effort or use fully synthetic generation prone to hallucination, and none offer scientifically grounded evaluation of LLMs on climate science QA across multiple task forms.
Approach
A textbook-grounded, RAG-plus-prompt-evolution generator (GPT-3.5-turbo) whose outputs are validated by climate scientists via a web UI, then used to train a GPT-4o-mini evaluator with uncertainty-based sampling to automate annotation; new metrics (GPT-4o-mini factual entailment score for freeform, context-window phrase similarity for cloze) are proposed for evaluation.
Key results
- GPT-4o with RAG on source textbooks reached 92.79% overall MCQ accuracy and 71.88 exact match on cloze on ClimaQA-Gold.
- RAG from source textbooks consistently outperformed continued pre-training and fine-tuning across tasks; retrieving from held-out books often reduced performance.
- About 85% of generated MCQs and 90% of freeform QAs were judged valid, and the evaluator model reportedly improved quality by 10% (MCQ) and 5% (freeform).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Retrieve from source textbooks (ablation) | GPT-4o · ClimaQA-Gold MCQ | Overall accuracy: 85.71 -> 92.79 | Helped |
| Retrieve from source textbooks (ablation) | GPT-4o · ClimaQA-Gold Cloze | Exact Match: 53.12 -> 71.88 | Helped |
| Retrieve from source textbooks (ablation) | GPT-4o · ClimaQA-Gold Freeform | Factual Accuracy (overall): 0.76 -> 0.83 | Helped |
| Retrieve from held-out textbooks (ablation) | GPT-4o · ClimaQA-Gold MCQ | Overall accuracy: 85.71 -> 81.90 | Hurt |
| Retrieve from held-out textbooks (ablation) | GPT-4o · ClimaQA-Gold Cloze | Exact Match: 53.12 -> 50.62 | Hurt |
| Few-shot prompting (ablation) | GPT-4o · ClimaQA-Gold MCQ | Overall accuracy: 85.71 -> 86.39 | Helped |
| Few-shot prompting (ablation) | GPT-4o · ClimaQA-Gold Freeform | BLEU (overall): 0.480 -> 0.448 | Hurt |
| Continued pre-train on held-out textbooks (ablation) | LLaMA3.1-8B · ClimaQA-Gold MCQ | Overall accuracy: 70.61 -> 75.92 | Helped |
| Continued pre-train on held-out textbooks (ablation) | Mistral-7B · ClimaQA-Gold MCQ | Overall accuracy: 67.76 -> 69.8 | Helped |
| Fine-tune on ClimaQA-Silver (ablation) | LLaMA3.1-8B · ClimaQA-Gold Cloze | Exact Match: 26.25 -> 51.25 | Helped |
| Fine-tune on ClimaQA-Silver (ablation) | Mistral-7B · ClimaQA-Gold Cloze | Exact Match: 17.50 -> 41.88 | Helped |
Surprising findings
- Most models struggled with reasoning-based MCQs, though they did better on reasoning freeform questions.
- Roughly 25% of GPT-3.5-turbo generated MCQs were answered incorrectly by the same model even when context was given.
Limitations
- Benchmark relies on only five source textbooks, limiting context diversity.
- Small expert-annotated set constrains the automated annotation evaluator.
- Textbook scraped data cannot be released due to copyright, hindering exact reconstruction.
Experimental setup
Datasets: ClimaQA-Gold ClimaQA-Silver Climate textbooks (18 graduate-level) Climate-Fever
Benchmarks: ClimaQA-Gold ClimaQA-Silver
Metrics: accuracy exact match phrase similarity BLEU ROUGE BERTScore factual accuracy
Baselines: gpt-3.5-turbo gpt-4o gemma-27b llama3-70b mixtral-8x22b gemini-1.5-flash mistral-7b llama3.1-8b
Builds on: Retrieval-Augmented Generation LoRA Chain-of-thought prompting Climate-Fever
Closest papers by meaning
- Assessing Large Language Models on Climate Information (ICML 2024 · similarity 0.90)
- AtmosSci-Bench: Evaluating the Recent Advance of Large Language Model for Atmospheric Science (NeurIPS 2025 · similarity 0.87)
- QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform (AAAI 2025 · similarity 0.85)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.84)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.84)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.