Research map / Large language models / Agents, code and math
Automated Statistical Model Discovery with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BoxLM, which uses LLMs (GPT-4) inside Box's Loop to propose, fit (via PyMC/JAX), and critique probabilistic programs for automated statistical model discovery.
Problem
Automating statistical model discovery, which normally requires expert-designed DSLs and handcrafted search procedures, over open-ended spaces of probabilistic models under domain and natural-language constraints.
Approach
An LM acts as modeler (proposing probabilistic programs given data, metadata, and exemplars) and as critic (summarizing fit statistics and posterior predictive summaries into natural-language feedback), iterated within Box's Loop, so no domain-specific language or hand-designed search is needed.
Key results
- BoxLM matches the Automatic Statistician's test MAE across six time series datasets in the restricted kernel DSL setting.
- BoxLM programs match expert Stan programs on 9 of 12 dataset/ablation settings by ELPD LOO (non-significant differences).
- In the Lotka-Volterra task, all BoxLM variants outperform standard Lotka-Volterra, neural ODE, and hybrid neural ODE baselines in test MAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove dataset metadata (ablation) | BoxLM · Surgical sim | ELPD LOO: -38.38 -> -63.72 | Hurt |
| remove dataset metadata (ablation) | BoxLM · Peregrine | ELPD LOO: -173.11 -> -136.39 | Helped |
| use simulated observations (ablation) | BoxLM · Dugongs | ELPD LOO: 23.40 -> 57.40 | Helped |
| use simulated observations (ablation) | BoxLM · Peregrine | ELPD LOO: -173.11 -> -179.06 | Hurt |
| use visual-only dataset and criticism inputs instead of textual inputs (ablation) | GPT-4 BoxLM · Peregrine | ELPD LOO: -164.69 -> -143.45 | Helped |
| use visual-only dataset and criticism inputs instead of textual inputs (ablation) | GPT-4 BoxLM · Eight schools | ELPD LOO: -30.17 -> -30.40 | No clear effect |
| use GPT-3.5 Turbo instead of GPT-4 (ablation) | BoxLM · Dugongs | ELPD LOO: 22.61 -> 21.01 | Hurt |
| use GPT-3.5 Turbo instead of GPT-4 (ablation) | BoxLM · Eight schools | ELPD LOO: -30.17 -> -30.44 | No clear effect |
| augment the base kernel space with additional kernels (ablation) | BoxLM · Australian beer sales | MAE | Helped |
| augment the base kernel space with additional kernels (ablation) | BoxLM · other time-series datasets | MAE | No clear effect |
| use the state-space criticism update instead of selecting top programs (ablation) | BoxLM · Stan datasets | ELPD LOO | No clear effect |
| add LM-driven corrections to the standard Lotka-Volterra model | Lotka-Volterra · predator-prey simulated data | test MAE | Helped |
Surprising findings
- Removing dataset metadata sometimes improved performance; on peregrine, the no-metadata run used a quartic polynomial and closed the gap with the expert model, since domain priors led to a logistic growth model that cannot capture the initial decline.
- On the surgical no-metadata ablation and dugongs no-metadata state-space variant, LM programs performed much worse than expert programs.
Limitations
- Only static, one-dimensional datasets were studied; criticism statistics were fixed in advance rather than automated.
- Some proposed PyMC programs fail inference (22% failure for GPT-4 textual, 30% for vision only, 24% for GPT-3.5).
- Performance is sensitive to domain metadata, which can bias the LM toward overly strong prior-driven choices.
Experimental setup
Datasets: Stan PosteriorDB eight schools Stan PosteriorDB dugongs Stan PosteriorDB surgical Stan PosteriorDB peregrine six univariate time series datasets (Automatic Statistician) simulated Lotka-Volterra perturbed dynamics
Benchmarks: Stan PosteriorDB eight schools, dugongs, surgical, peregrine (and simulated / no-metadata variants) six univariate time series datasets (GP kernel search) perturbed Lotka-Volterra test extrapolation
Metrics: ELPD LOO mean absolute error squared error
Baselines: Automatic Statistician periodic kernel GP spectral mixture kernel GP N-BEATS expert Stan programs (PosteriorDB) standard Lotka-Volterra neural ODE hybrid neural ODE
Builds on: Automatic Statistician Box's Loop PyMC Jax diffrax
Closest papers by meaning
- Automated Model Discovery via Multi-modal & Multi-step Pipeline (NeurIPS 2025 · similarity 0.87)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.86)
- Explaining Datasets in Words: Statistical Models with Natural Language Parameters (NeurIPS 2024 · similarity 0.86)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.85)
- Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context (NeurIPS 2025 · similarity 0.85)
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.