Research map / Probabilistic and scientific modeling / Variational and generative sampling
Ensembling Language Models with Sequential Monte Carlo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified framework of generalized-mean (𝔣-)ensembles over strings for combining language models and a byte-level SMC algorithm to sample them, showing consensus-seeking ensembles (min, product) beat probability averaging on structured generation tasks.
Problem
Naive token-level ensembling of language models samples from a locally normalized, biased approximation of the intended string-level ensemble distribution, and mismatched tokenizers complicate aggregation.
Approach
Define string-level ensembles as generalized means of expert potentials (derived as minimizers of weighted alpha-divergences), map all models to a shared byte/character space, and sample from the global ensemble with a byte-level sequential Monte Carlo algorithm using the ensemble function as a shaping function.
Key results
- Best global consensus ensemble (token SMC, M=25) reaches 87.8% on JSON schema for Phi vs 83.5% best single prompt.
- On BBH word sorting, cross-model Phi+Llama byte-SMC ensemble reaches 41.1% vs single-prompt baseline around 38.6% for Llama.
- Consensus-seeking operators (min, product) show medium positive Spearman correlation between approximation quality and expected accuracy, while sum and max show weak negative correlations.
Surprising findings
- Coverage-seeking probability averaging can underperform the best single model, since its expected accuracy is bounded by the average of base models.
- Better posterior approximation (higher E[log Z]) does not always improve accuracy; coverage-seeking ensembles show weak negative correlation.
Limitations
- Only two experts evaluated, limiting analysis of multi-model ensembles.
- Substantial inference-time overhead from SMC particles and byte-level marginalization over tokenizations.
- Evaluated only on structured binary-correctness tasks with 7-14B instruction-tuned models.
Experimental setup
Datasets: GlaiveAI function-calling dataset JSONSchemaBench BIG-Bench Hard word sorting SPIDER
Benchmarks: JSONSchemaBench BIG-Bench Hard (word sorting) SPIDER
Metrics: expected accuracy under ensemble (SMC estimate) execution accuracy binary correctness Spearman correlation with log marginal likelihood
Baselines: Local Prob. Avg. best single prompt best local f-ensemble token-level SMC
Builds on: sequential importance sampling sequential monte carlo steering alpha-divergence product of experts
Closest papers by meaning
- Rethinking LLM Ensembling from the Perspective of Mixture Models (ICML 2026 · similarity 0.91)
- Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling (ICLR 2025 · similarity 0.91)
- Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration (NeurIPS 2024 · similarity 0.89)
- Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo (ICLR 2025 · similarity 0.89)
- Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo (ICML 2024 · similarity 0.88)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.