Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Architectural and Inferential Inductive Biases for Exchangeable Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that multi-step autoregressive inference beats one-step inference for uncertainty quantification and decision-making with exchangeable sequence models, and that masking-based permutation-invariant transformers do not guarantee exchangeability and underperform causal masks.
Problem
Autoregressive transformers for exchangeable sequences are typically used with one-step predictions, which cannot separate epistemic from aleatoric uncertainty; prior masking-based architectures are assumed to enforce exchangeability without verification.
Approach
Argues for multi-step autoregressive generation to recover posterior/epistemic uncertainty via De Finetti, and formalizes "conditional permutation invariance" as weaker than exchangeability, showing masking-based transformers violate the c.i.d. (martingale) property and perform worse than standard causal masks.
Key results
- Multi-step inference yields up to 60% lower Bayesian regret than one-step inference in Thompson sampling bandits.
- Multi-step uncertainty sampling reaches the same active learning performance with nearly 10x fewer samples than one-step inference.
- Standard causal masking outperforms the conditionally permutation-invariant masking by about 10% out of training horizon and up to 20% in training efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-step instead of one-step inference | Transformer-based uncertainty quantification · Synthetic GP | multi-step log-loss | Helped |
| Use multi-step instead of one-step inference | Transformer with Thompson sampling · Bayesian multi-armed bandits | Bayesian regret: up to $60\%$ less regret | Helped |
| Use multi-step instead of one-step inference | Transformer with uncertainty sampling · Active learning | samples to achieve the same predictive performance: nearly $10$ times fewer samples | Helped |
| Use conditionally permutation-invariant rather than standard causal masking | Decoder-only Transformer · Synthetic GP, in-training horizon | multi-step log-loss | No clear effect |
| Use standard causal rather than conditionally permutation-invariant masking | Decoder-only Transformer · Synthetic GP, out-of-training horizon | multi-step log-loss: approximately $10\%$ | Helped |
| Use standard causal rather than conditionally permutation-invariant masking | Decoder-only Transformer · Synthetic GP, training/data efficiency | multi-step log-loss: up to $20\%$ | Helped |
Surprising findings
- Enforcing conditional permutation invariance via masking does not guarantee exchangeability and performs worse than standard causal masking.
- The permutation-invariant masking scheme cannot benefit from KV caching, giving higher inference compute than causal masking.
Limitations
- Experiments are limited to small synthetic settings (GP, Gaussian bandits, synthetic regression) with a small 4-layer transformer.
- Theoretical bandit result is shown for a specific one-armed Gaussian scenario only.
Experimental setup
Datasets: synthetic Gaussian process data (RBF kernel) synthetic Gaussian bandit rewards synthetic heteroscedastic regression with 100 clusters
Benchmarks: synthetic GP multi-step log-loss Bayesian bandit regret (Gaussian arms) active learning test log-likelihood on synthetic regression
Metrics: multi-step log-loss one-step log-loss Bayesian regret test negative log-likelihood
Baselines: one-step inference standard causal masking Thompson sampling with one-step inference
Builds on: De Finetti's theorem Thompson sampling Prior-data fitted networks Transformer neural processes
Closest papers by meaning
- Model-Informed Flows for Bayesian Inference (NeurIPS 2025 · similarity 0.85)
- Autoregressive Learning in Joint KL: Sharp Oracle Bounds and Lower Bounds (NeurIPS 2026 · similarity 0.85)
- Asymptotically Log-Optimal Bayes-Assisted Confidence Sequences for Bounded Means (NeurIPS 2026 · similarity 0.85)
- Amortizing intractable inference in large language models (ICLR 2024 · similarity 0.85)
- SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking (ICLR 2024 · similarity 0.85)
- Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.