Research map / Vision-language and multimodal / Robot action and navigation
VESTA: Visual Exploration with Statistical Tool Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VESTA, a VLM agent framework that dynamically creates and reuses visual diagnostic tools to refine PyMC statistical models, and DAWN, a synthetic and astronomy benchmark for distribution fitting and time-series modeling.
Problem
Automating the fitting of quantitative probabilistic models to data is hard; existing LLM/VLM agent systems struggle on complex distributions and time series, relying on iterative critique without exploring the data with diagnostic tools.
Approach
A VLM agent that, at each refinement step, either selects an existing visual diagnostic tool from a growing registry or writes a new Python tool, whose visual outputs feed a VLM critic that proposes revised PyMC models scored by AIC; tools persist and are reused within a run.
Key results
- On DAWN Hard distribution fitting, VESTA-Dynamic beats PyVision and BoxLM; Claude Sonnet 4.6 VESTA-Expert reaches JS gap -0.136 vs PyVision (p<0.001 after Holm correction).
- Time-series ELPD-LOO: VESTA variants beat PyVision in all 9 (LLM, split) cells, with significant gains such as +251 nats over PyVision for Claude on Hard (VESTA-Dynamic).
- Generated tools compose on average 3.5 expert tools per tool versus 1.9 for PyVision; 79.5% of dynamically generated tools are re-invoked within a problem instance.
Surprising findings
- Accumulating tools across 50 training runs yields a narrow toolkit (4 tools) that performs worst among the tool conditions, so on-the-fly generation at test time is the main driver.
- Dynamic visual tools do not outperform expert-written tools, which the authors attribute to VLM difficulty parsing complex multi-panel diagrams.
Limitations
- Dynamically generated visual tools underperform expert tools, likely because VLMs struggle to read complex multi-panel diagrams.
- Several Box-LM comparisons on Claude are not significant due to high posterior variance of the baseline.
- Distribution-fitting results on Easy and Astro splits are not statistically significant after correction.
Experimental setup
Datasets: DAWN
Benchmarks: DAWN
Metrics: Jensen-Shannon divergence ELPD-LOO AIC expert tool coverage
Baselines: BoxLM PyVision VESTA-None VESTA-Expert
Builds on: Box's Loop PyVision BoxLM PyMC
Closest papers by meaning
- AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy (NeurIPS 2025 · similarity 0.88)
- A systematic evaluation of vision-language models for observational astronomical reasoning tasks (NeurIPS 2026 · similarity 0.86)
- The Perception–Physics Paradox: Probing Scientific Alignment with TC-Bench (ICML 2026 · similarity 0.83)
- Mimicking the Physicist's Eye: A VLM-centric Approach for Physics Formula Discovery (NeurIPS 2026 · similarity 0.83)
- AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ (ICLR 2024 · similarity 0.83)
- DiSciPLE: Learning Interpretable Programs for Scientific Visual Discovery (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.