Research map / Large language models / Language, safety and interpretability
Language Model Cascades: Token-Level Uncertainty And Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies deferral rules for LM cascades, showing sequence-level probability suffers from length bias and proposing learned post-hoc deferral rules over token-level probability quantiles (plus optional intermediate embeddings) for FLAN-T5 Base→Large cascades.
Problem
Deciding when a small generative LM should defer to a large one in a cascade; naive sequence-probability confidence (Chow's rule extended to sequences) suffers from length bias and ignores token-level uncertainty.
Approach
Replace sequence-level probability with quantiles of per-token probabilities from the small model as features, and learn a post-hoc deferral scorer (MLP) trained on labels indicating whether the large model would do better; optionally add embeddings from the small model's decoder and the large model's intermediate layer.
Key results
- Post-Hoc-Embed-1+2 achieves the best AUC-DF on 12 of 17 datasets, e.g., MNLI 0.722 (+12.24%) and TriviaQA 0.100 (+19.56%) over the random baseline.
- Chow-Sum yields negative gains on MNLI (-2.63%) and TriviaQA (-13.09%), while Chow-Average over-corrects, so no fixed quantile works across all datasets.
- Post-Hoc-Quantile is the most consistently performant non-embedding method, reaching 0.711 on MNLI and 0.097 on TriviaQA AUC-DF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned post-hoc token-quantile features instead of Chow-Sum | FLAN-T5 Base → Large cascade · MNLI | AUC-DF: 0.627 -> 0.711 | Helped |
| Use learned post-hoc token-quantile features instead of Chow-Sum | FLAN-T5 Base → Large cascade · TriviaQA | AUC-DF: 0.073 -> 0.097 | Helped |
| Use learned post-hoc token-quantile features instead of Chow-Sum | FLAN-T5 Base → Large cascade · WMT DE → FR | AUC-DF: 0.390 -> 0.404 | Helped |
| Add large-model intermediate decoder embedding to the post-hoc rule using small-model decoder embeddings | FLAN-T5 Base → Large cascade · MNLI | AUC-DF: 0.711 -> 0.722 | Helped |
| Add large-model intermediate decoder embedding to the post-hoc rule using small-model decoder embeddings | FLAN-T5 Base → Large cascade · TriviaQA | AUC-DF: 0.097 -> 0.100 | Helped |
| Add large-model intermediate decoder embedding to the post-hoc rule using small-model decoder embeddings | FLAN-T5 Base → Large cascade · WMT DE → FR | AUC-DF: 0.404 -> 0.404 | No clear effect |
| Add large-model intermediate decoder embedding to the post-hoc rule using small-model decoder embeddings | FLAN-T5 Base → Large cascade · WMT FR → EN | AUC-DF: 0.619 -> 0.618 | No clear effect |
| Use Chow-Quantile-0.8 instead of Chow-Sum | FLAN-T5 Base → Large cascade · MNLI | AUC-DF: 0.627 -> 0.691 | Helped |
| Use Chow-Average instead of Chow-Sum | FLAN-T5 Base → Large cascade · TriviaQA | AUC-DF: 0.073 -> 0.087 | Helped |
| Include encoder embeddings in the post-hoc embedding methods (ablation) | FLAN-T5 Base → Large cascade · MNLI, TriviaQA, WMT DE → FR | AUC-DF | No clear effect |
| Use token-probability quantiles instead of a padded probability vector (ablation) | Post-hoc deferral rule · Multiple datasets | AUC-DF | Helped |
| Use token-probability quantiles instead of a sorted padded probability vector (ablation) | Post-hoc deferral rule · Multiple datasets | AUC-DF | Helped |
Surprising findings
- Naively feeding the full probability vector padded with zeros performs poorly compared to quantile features.
- Chow-Average over-corrects the length bias and defers shorter outputs, so simple length normalization is not a fix.
Limitations
- Experiments mainly use encoder-decoder FLAN-T5 models with few-shot greedy decoding; decoder-only and RLHF models are not studied.
- No single fixed quantile works across datasets, requiring learned combination.
- Intermediate embedding methods incur extra cost from extracting large-model decoder embeddings.
Experimental setup
Datasets: MNLI TriviaQA WMT DE→FR IMDb SuperGLUE ANLI NaturalQA TyDiQA Lambada SQuAD WMT14 EN→FR WMT19 DE→FR
Benchmarks: MNLI TriviaQA WMT DE→FR ANLI-R1 ANLI-R2 ANLI-R3 BoolQ IMDb Lambada PIQA SQuAD TyDiQA-FI
Metrics: AUC-DF (area under deferral curve) AUC-ROC for golden deferral label
Baselines: Chow-Sum Chow-Average Chow-Quantile-0 Chow-Quantile-0.4 Chow-Quantile-0.8 Post-Hoc-Quantile
Builds on: Chow's rule FrugalGPT T5 FLAN-T5
Closest papers by meaning
- Forking Paths in Neural Text Generation (ICLR 2025 · similarity 0.90)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.89)
- SPANUQ: Span-Level Uncertainty Quantification for Large Language Model Generation (NeurIPS 2026 · similarity 0.89)
- Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.88)
- Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.88)
- Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.