Research map / Large language models / AI and society
AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AstroVisBench, an 864-task benchmark (from 110 astronomy Jupyter notebooks) testing LLMs on processing code and expert-validated VLM-judged visualization generation in astronomy.
Problem
Evaluating whether LLM-generated scientific workflows in astronomy produce visualizations that convey correct scientific insights, including domain-specific library use, has not been addressed by prior benchmarks.
Approach
Builds execution-based evaluation from notebook dependencies: processing tasks are scored by recall of key intermediate variables (VIscore), and visualization tasks are scored by a Claude 3.5 Sonnet VLM judge validated against five professional astronomers' error ratings.
Key results
- Best processing VIscore is 0.694 (o3-mini); Gemini 2.5 Pro has the lowest processing crash rate at 30.8%.
- Claude 3.5 Sonnet judge achieves Spearman rho 0.822 (avg) with expert judgments, the highest among tested VLM judges.
- Gemini 2.5 Pro yields the most no-error visualizations (15.7%) but still has 28.5% major errors; around 43% of all generated code fails to execute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini 2.5 Pro instead of the other evaluated models | Astronomy processing code generation · AstroVisBench processing | Crash (%): 30.8 | Helped |
| Use o3-mini instead of the other evaluated models | Astronomy processing code generation · AstroVisBench processing | VIscore: 0.694 | Helped |
| Use Gemini 2.5 Pro instead of the other evaluated models | Astronomy visualization code generation · AstroVisBench visualization | Crash (%): 20.1 | Helped |
| Use o3-mini instead of the other evaluated models | Astronomy visualization code generation · AstroVisBench visualization | VisFail (%): 2.8 | Helped |
| Use Gemini 2.5 Pro instead of the other evaluated models | Astronomy visualization code generation · AstroVisBench visualization | NoE (%): 15.7 | Helped |
| Use QwQ instead of the other evaluated models | Astronomy visualization code generation · AstroVisBench visualization | MaE (%): 14.9 | Helped |
| Use Gemini 2.5 Pro for execution success rather than assuming execution success implies correct processing | Astronomy processing code generation · AstroVisBench processing | VIscore: 0.600 vs. 0.694 | Mixed |
Surprising findings
- Higher code execution success does not imply better scientific computation: Gemini 2.5 Pro runs most often but has a lower VIscore than o3-mini.
- Gemini 2.5 Pro, the best model, still has at least 58% of visualization outputs that either fail or contain major errors.
Limitations
- Benchmark queries and clarifications are generated by GPT-4o with only partial expert verification, so hallucination noise may remain.
- VLM judge may diverge from expert judgments; agreement computed on only 30 tasks.
- Variable inspection is limited because some runtime objects cannot be pickled.
Experimental setup
Datasets: Astro Data Lab notebooks STScI notebooks
Benchmarks: AstroVisBench
Metrics: VIscore crash rate VisFail rate no error rate minor error rate major error rate Spearman correlation with expert judgments Fleiss kappa
Baselines: Gemini 2.5 Pro Claude 3.7 Sonnet Claude 4.0 Opus o3-mini GPT-4o QwQ 32B Qwen 2.5 72B Llama-4 Maverick
Builds on: SWE-bench BigCodeBench VisEval EvaLLM
Closest papers by meaning
- A systematic evaluation of vision-language models for observational astronomical reasoning tasks (NeurIPS 2026 · similarity 0.89)
- VESTA: Visual Exploration with Statistical Tool Agents (NeurIPS 2026 · similarity 0.88)
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.86)
- LTD-Bench: Evaluating Large Language Models by Letting Them Draw (NeurIPS 2025 · similarity 0.84)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.84)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.