Research map / Vision-language and multimodal / Multimodal LLMs

FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FUSE, a training-free-at-inference Bayesian framework that fuses GP-adapted CLIP embedding uncertainty (prior) with a log-det semantic evidence statistic from sampled LLaVA responses (likelihood) into a closed-form posterior for VQA uncertainty and selective answering.

Problem

Quantifying uncertainty of vision-language model outputs so that the score predicts answer correctness and supports selective answering, combining input-level ambiguity with model-level response variability.

Approach

Treats input-embedding dispersion from GP-probabilistic CLIP adapters as a Gaussian prior over a latent uncertainty level u, and the log-determinant of the Gram/scatter matrix of sampled response embeddings as linear-Gaussian evidence in u; a closed-form Bayesian update yields a fused posterior score u* = m_post + z*sigma_post.

Key results

• FUSE achieves AUROC 0.883 on VQAv2 versus 0.862 for the best baseline UMPIRE, and 0.761 vs 0.750 on OKVQA and 0.796 vs 0.773 on AdVQA.
• FUSE reports ECE of 0.037 on VQAv2, 0.037 on OKVQA and 0.041 on AdVQA, the lowest among the compared methods.
• FUSE reports AURAC of 0.940 on VQAv2 vs 0.916 for UMPIRE; the paper claims consistent gains across datasets.

Surprising findings

• Global Tuning of the posterior parameters remains a strong ablation baseline even though parameters are learned, which the authors take as evidence for the value of using both sources.

Limitations

• Gaussian and linear-Gaussian approximations rely on asymptotic and heuristic arguments (e.g., CLT in d, approximate remainder terms for weighted case).
• Evaluated only on VQA with exact-match correctness and a single VLM (LLaVA-v1.5-13B).

Experimental setup

Datasets: MS-COCO Flickr30k VQAv2

Benchmarks: VQAv2 OKVQA AdVQA

Metrics: AUROC TPR at 0.1/0.05/0.01 FPR Pearson correlation Spearman correlation ECE AURAC

Baselines: UMPIRE Neighborhood Consistency LN-Entropy Semantic Entropy EigenScore LoFreeCP CLM

Builds on: UMPIRE Sparse variational GPLVM EigenScore CLIP

Closest papers by meaning

• CertainlyUncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness (ICLR 2025 · similarity 0.91)
• Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification (ICLR 2026 · similarity 0.90)
• Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding (NeurIPS 2025 · similarity 0.89)
• Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.89)
• CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation (NeurIPS 2026 · similarity 0.89)
• Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.87)