Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Position: The Future of Bayesian Prediction Is Prior-Fitted

ICML 2025 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Approximate inference in Bayesian neural networks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper arguing that Prior-Data Fitted Networks (PFNs) and related amortized inference methods will dominate many Bayesian prediction applications, and outlining their scope, limitations, and research directions.

Problem

Bayesian prediction typically requires per-dataset inference (MCMC, VI, GPs) that is slow, hard to implement, and restricted to tractable priors, while real-world data is not growing as fast as compute.

Approach

Argues that PFNs, which pre-train transformers on synthetic datasets from a declaratively specified prior to approximate the posterior predictive distribution in-context, should become the main approach for Bayesian prediction; proposes extensions such as in-context interpreters for probabilistic programs, RL-based Bayesian optimization, latent prediction, and inference speedups.

Key results

• Small-scale experiment with a GP prior shows predictive standard deviation shrinks in early roll-out steps before stabilizing, a partial violation of the Martingale property, with average standard deviation of 0.21.
• Reports PFNs over 200x faster than previous methods on small-scale Bayesian prediction and over 10,000x faster for learning curve extrapolation (cited prior work).
• TabPFN v2 pre-training reported at 2,700 GPU hours.

Measured effects reported in the paper

Surprising findings

• Original PFN architecture fails to count repeated identical training examples, so its predictions stay static as evidence accumulates in a coin-flip prior.

Limitations

• PFNs currently work best on smaller datasets and are commonly outperformed on large datasets.
• Quadratic attention scaling limits sample size; inference is slow relative to traditional tabular methods.
• PFNs struggle with heterogeneous feature distributions such as mixing centered and heavy-tailed features; they hide latents, reducing interpretability.

Experimental setup

Datasets: synthetic datasets from GP prior synthetic coin-flip prior synthetic linear regression prior

Metrics: standard deviation of predictions across roll-out steps training loss

Baselines: MCMC variational inference Gaussian processes XGBoost

Builds on: Prior-Data Fitted Networks TabPFN Neural Processes Simulation-based inference

Closest papers by meaning

• Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks (ICML 2025 · similarity 0.87)
• Do-PFN: In-Context Learning for Causal Effect Estimation (NeurIPS 2025 · similarity 0.86)
• Do Bayesian Neural Networks Actually Behave Like Bayesian Models? (ICML 2025 · similarity 0.86)
• A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.86)
• Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models (NeurIPS 2025 · similarity 0.86)
• Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality (NeurIPS 2025 · similarity 0.85)