Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

What does a Bayes-filtered transformer believe? A predictive Monte Carlo approach

NeurIPS 2026 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Simulation-based inference · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes predictive Monte Carlo (PMC) as a black-box latent-space interpretability tool for Bayes-filtered transformers, recovering implicit priors and posteriors over latent tasks from next-token rollouts across balls-and-urns, linear regression, and Markov chain families.

Problem

Prediction-space comparisons cannot reliably tell which prior and posterior over the latent task a trained Bayes-filtered transformer has internalized, since distinct posteriors can yield identical predictions.

Approach

Apply predictive Monte Carlo to any trained BFT: roll out next tokens from an empty or given prompt and compute the latent task (empirical frequency, OLS fit, or empirical transition matrix) from each completed path, yielding samples of the implicit prior and posterior in latent space using only generation.

Key results

• PMC prior marginals match the training prior Pi_M for linear regression at M=8 and the posterior matches the memorizing posterior on in-distribution prompts.
• The task-diversity threshold appears in latent space: as M grows the PMC posterior moves from the memorizing toward the generalizing baseline across families.
• Transient generalization is reproduced in latent space at intermediate task diversity (e.g., M=32 for balls-and-urns and linear regression, M=8 for Markov chains).

Surprising findings

• On out-of-distribution prompts in linear regression, PMC posterior samples match neither the memorizing nor the generalizing posterior.
• For balls-and-urns, the task-diversity transition does not appear in prediction space at the prompt length used, but does appear in latent space.

Limitations

• PMC validity rests on asymptotic exchangeability / recurrence conditions that are not verified for trained BFTs.
• Mismatch on out-of-distribution linear regression prompts cannot be attributed to model departure versus PMC invalidity.
• Whether PMC transfers to LLMs is unknown.

Experimental setup

Datasets: synthetic balls-and-urns sequences synthetic linear regression sequences synthetic Markov chain sequences

Benchmarks: balls-and-urns synthetic task linear regression synthetic task Markov chain synthetic task

Metrics: squared energy distance sliced Wasserstein-1 symmetrized KL MSE gap

Baselines: memorizing posterior generalizing posterior

Builds on: predictive resampling prior-data fitted networks de Finetti representation theorem Diaconis-Freedman representation theorem

Closest papers by meaning

• Bayesian Additive Distribution Regression (NeurIPS 2026 · similarity 0.83)
• HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.83)
• Can Transformers Learn Full Bayesian Inference in Context? (ICML 2025 · similarity 0.83)
• Transformers on Markov data: Constant depth suffices (NeurIPS 2024 · similarity 0.83)
• Constrained Belief Updates Explain Geometric Structures in Transformer Representations (ICML 2025 · similarity 0.83)
• Towards Understanding the Universality of Transformers for Next-Token Prediction (ICLR 2025 · similarity 0.83)