Research map / Vision-language and multimodal / Multimodal LLMs

A systematic evaluation of vision-language models for observational astronomical reasoning tasks

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AstroVLBench, a 4,100+ instance benchmark across five astronomical observational modalities, and evaluates six frontier VLMs with prompt, representation, few-shot and reasoning-quality ablations.

Problem

Reliability of general-purpose vision-language models on real multi-modal astronomical observations (imaging, radio, SED plots, light curves, spectra) is untested; it is unclear whether failures stem from perception, missing domain knowledge, or flawed reasoning.

Approach

An AGN-centered benchmark across five observational modalities with controlled ablations (guided vs phenomenological vs physical prompts, plots vs numerical tables, few-shot exemplars) and a qualitative audit of reasoning validity to detect right-answer-wrong-reason cases.

Key results

• Gemini 3 Pro achieves 74.5% on Task 1 (QSO host, N=557) and leads four of five tasks, e.g., 61.3% on Task 4 light curves versus 38.7% for GPT-5.2.
• Physical prompts raise Task 1 accuracy to 74.5% from 64.1% unguided with more balanced class recall (76%/73%).
• Numerical tables improve Task 4 light curve accuracy to 74.6% versus 61.3% for plots (+13.3 pp).

Surprising findings

• Phenomenological prompting beats physical prompting on Task 4 (65.5% vs 61.3%), driven by RR Lyrae recall, suggesting physical prompts can suppress relevant observational reasoning.
• Several frontier models suffer complete mode collapse (e.g., predicting one class for all inputs) despite high aggregate accuracy on imbalanced tasks.

Limitations

• Small sample sizes for some tasks (e.g., N=142 light curves, N=168 SED), limiting statistical power.
• Reasoning-quality analysis is a qualitative review of Task 5 examples rather than systematic scoring.
• Only API-accessed models evaluated at temperature 0; no fine-tuned or open-weight VLM training.

Experimental setup

Datasets: HSC-SSP PDR3 MiraBest FIRST NVSS AKARI NEP PLAsTiCC DESI DR1

Benchmarks: AstroVLBench

Metrics: accuracy 95% bootstrap confidence interval per-class recall specificity

Baselines: Gemini 3 Pro GPT-5.2 Claude Opus 4.5 Grok-4 Qwen3-235B Intern-S1-Pro

Builds on: MiraBest PLAsTiCC FastSpecFit

Closest papers by meaning

• AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy (NeurIPS 2025 · similarity 0.89)
• VESTA: Visual Exploration with Statistical Tool Agents (NeurIPS 2026 · similarity 0.86)
• TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.86)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.85)
• VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing (NeurIPS 2026 · similarity 0.85)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs (NeurIPS 2025 · similarity 0.85)