Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the first bias-variance-covariance decomposition for kernel scores over generative model predictive distributions and gives sample-only unbiased, consistent estimators, applied to image, audio and LLM uncertainty estimation.
Problem
Generative models lack a theoretical framework for generalization and uncertainty estimation; existing bias-variance decompositions cover only MSE or require access to predictive distributions, which closed-source models do not expose.
Approach
Extends the bias-variance-covariance decomposition from MSE to kernel scores, defining distributional variance and covariance via a kernel inner product, and estimates them only from generated samples; predictive kernel entropy (negative squared kernel norm of a single model) serves as a single-model uncertainty measure.
Key results
- On TriviaQA and CoQA, kernel entropy (RBF, e5-small-v2 embeddings) outperforms baselines such as semantic entropy and lexical similarity in AUROC across OPT model sizes in almost all cases.
- Distributional variance correlates with squared MMD at Pearson approx. 0.95 throughout diffusion model training on InfiMNIST.
- Mode collapse of digit '0' (reduced to about 1% of training data) appears only in the bias term, while variance keeps decreasing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CS kernel entropy instead of lexical similarity | OPT question-answering models · TriviaQA | AUROC: 0.520 vs 0.582; 0.602 vs 0.577; 0.657 vs 0.656; 0.653 vs 0.648; 0.702 vs 0.668; 0.692 vs 0.644 | Mixed |
| Use CS kernel entropy instead of lexical similarity | OPT question-answering models · CoQA | AUROC: 0.593 vs 0.535; 0.623 vs 0.591; 0.678 vs 0.629; 0.700 vs 0.618; 0.710 vs 0.638; 0.700 vs 0.641 | Helped |
| Use a sequence kernel rather than an embedding-based kernel (ablation) | Kernel entropy for LLM answer uncertainty · CoQA and TriviaQA | AUROC | Hurt |
| Use kernel entropy instead of lexical similarity without a pretrained semantic model | Closed-source LLM uncertainty estimation · CoQA and TriviaQA | AUROC: TriviaQA: 0.520 vs 0.582; 0.602 vs 0.577; 0.657 vs 0.656; 0.653 vs 0.648; 0.702 vs 0.668; 0.692 vs 0.644; CoQA: 0.593 vs | Helped |
| Increase the number of generated answers used to estimate kernel entropy (ablation) | OPT-13b · CoQA | AUROC | Helped |
| Replace the RBF kernel with a Laplacian kernel (ablation) | Embedding-based kernel entropy · CoQA and TriviaQA | AUROC | Hurt |
| Replace the RBF kernel with cosine similarity or a polynomial kernel (ablation) | Embedding-based kernel entropy · CoQA and TriviaQA | AUROC | No clear effect |
| Use an alternative text embedder (ablation) | RBF-kernel entropy · CoQA | AUROC | No clear effect |
| Use all-MiniLM-L6-v2 or all-MiniLM-L12-v2 as the embedder (ablation) | RBF-kernel entropy · TriviaQA | AUROC | Helped |
| Use the RBF kernel with audio vectors rather than the Laplacian kernel (ablation) | Glow-TTS · LJSpeech | Pearson correlation | Hurt |
| Use a Laplacian or polynomial kernel instead of the RBF kernel (ablation) | Conditional diffusion models · InfiMNIST | Generalization trends | No clear effect |
Surprising findings
- Mode collapse of an underrepresented class is expressed purely in the bias, with variance still decreasing, suggesting it is a contrary phenomenon to overfitting.
- For Glow-TTS audio, the kernel entropy vs. kernel score correlation is negative because badly fitted instances have narrower predicted distributions.
Limitations
- Bias and variance estimates depend on approximating the training distribution, e.g. via deep ensembles, which may not reflect ground truth.
- Estimators require i.i.d. samples; this may not hold for variance and covariance estimates across models.
- Language experiments are empirical and adopted from prior work, with no guarantees for new settings.
Experimental setup
Datasets: InfiMNIST LJSpeech CelebA CoQA TriviaQA
Benchmarks: CoQA TriviaQA InfiMNIST LJSpeech CelebA
Metrics: AUROC Pearson correlation kernel score MMD^2 bias variance
Baselines: predictive entropy normalised predictive entropy p(True) semantic entropy lexical similarity
Builds on: bias-variance-covariance decomposition (MSE) kernel scores maximum mean discrepancy semantic entropy Glow-TTS
Closest papers by meaning
- Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities (NeurIPS 2024 · similarity 0.89)
- Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.87)
- Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models (NeurIPS 2025 · similarity 0.86)
- Language Model Cascades: Token-Level Uncertainty And Beyond (ICLR 2024 · similarity 0.86)
- Towards a Scalable Reference-Free Evaluation of Generative Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.