Research map / Vision-language and multimodal / Multimodal LLMs
Automated Model Discovery via Multi-modal & Multi-step Pipeline
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-modal, multi-step agentic pipeline for automated model discovery in which a VLM analyzer proposes Gaussian process kernel compositions and a VLM evaluator scores fit and extrapolation from plots, combined with BIC into a Visual Information Criterion (VIC).
Problem
Automated model discovery must search a large combinatorial model space while capturing fine details and generalizing beyond the training region, and traditional criteria like BIC often fail to reward extrapolation plausibility.
Approach
Replace human expert roles with two VLM agents: AnalyzerVLM runs multi-step code-executing analysis (with plots as observations) to propose kernel candidates and initial hyperparameters, and EvaluatorVLM scores visual fitness and visual generalizability; these are combined with BIC into VIC for model selection.
Key results
- On the seven real-world univariate time series, Ours (GPT-4o-mini) achieves the best average test RMSE of 0.1070 vs 0.3121 for GP (SE), 0.3017 for Automatic Statistician, and 0.3474 for BoxLM, with train average 0.0134.
- Ours (GPT-4o) reaches average test RMSE 0.1556 and Ours (Qwen2.5-VL) 0.1122, both still below the baselines on average.
- Visual fitness and visual generalizability from EvaluatorVLM are reported to correlate with likelihood, generalization gap, and human ratings; the paper states the test MSE decreases over rounds with VIC while it increases with BIC alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full pipeline with AnalyzerVLM and EvaluatorVLM instead of a simpler proposal approach without AnalyzerVLM or BIC-only selection without EvaluatorVLM (ablation) | Automated model discovery pipeline · Gaussian Process Kernel Discovery | RMSE | Helped |
| Use visual representations for proposal and evaluation instead of text-only proposal and BIC-only evaluation (ablation) | Automated model discovery pipeline · Gaussian Process Kernel Discovery | MSE | Helped |
| Use multi-step AnalyzerVLM analysis instead of restricting AnalyzerVLM to a single step (ablation) | AnalyzerVLM · Gaussian Process Kernel Discovery | MSE | Helped |
| Use VLM rather than LLM as the analyzer (ablation) | Analyzer · Gaussian Process Kernel Discovery | analysis steps required | Helped |
| Use VIC instead of BIC alone for model selection (ablation) | Automated model discovery pipeline · Airline and Radio extrapolation | test MSE | Helped |
| Use AnalyzerVLM-proposed parameter initialization instead of random initialization (ablation) | Gaussian Process Kernel Discovery · Kernel hyperparameter optimization | model fit | Helped |
| Use GPT-4o instead of GPT-4o-mini as the VLM backbone (ablation) | Automated model discovery pipeline · Average Test | RMSE: 0.1070 -> 0.1556 | Hurt |
| Use Qwen2.5-VL instead of GPT-4o-mini as the VLM backbone (ablation) | Automated model discovery pipeline · Average Test | RMSE: 0.1070 -> 0.1122 | Hurt |
Surprising findings
- BIC-selected models often show deviations or unnatural flattening in the extrapolated region, which VIC penalizes.
- Symbolic regression methods underperform on real-world data because they do not explicitly model observation noise, whereas GP models do.
Limitations
- Current pipeline focuses on 1D datasets; multivariate extension is future work.
- Performance may depend on the quality of input visualizations.
- Experiments take multiple hours on 16-core CPU and rely on API VLMs with stochastic outputs.
Experimental setup
Datasets: gpss-research real-world univariate time series (Airline, Solar, Mauna Loa, Wheat, Call-Center, Radio, Gas) Nguyen Constant R Keijzer symbolic regression benchmarks
Benchmarks: Airline Solar Mauna Loa Wheat Call-Center Radio Gas Nguyen Constant R Keijzer
Metrics: RMSE R-square likelihood correlation generalization gap correlation
Baselines: Gaussian Process (SE) ARIMA Facebook Prophet Automatic Statistician BoxLM SGA LLM-SR ICSR
Builds on: Automatic Statistician BoxLM LLM-SR WaitGPT GPy-ABCD
Closest papers by meaning
- Automated Statistical Model Discovery with Language Models (ICML 2024 · similarity 0.87)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.86)
- Towards Robust Multi-Modal Reasoning via Model Selection (ICLR 2024 · similarity 0.85)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.84)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.84)
- GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.