Research map / Vision-language and multimodal / Visual question answering

CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation

NeurIPS 2026 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoMet, a generation-free post-hoc uncertainty estimator for MLLMs that decomposes uncertainty via a matching variable into a context-specific term and a multiplicity-specific term, using Rényi-2 entropy and lightweight heads on frozen Qwen3VL and InternVL3.5 backbones.

Problem

Estimating predictive uncertainty for multimodal LLMs in open-ended settings, where answer spaces are unbounded and existing methods rely on repeated generation, verbalized confidence, or fine-tuning.

Approach

Introduce a binary matching variable between input, context and answer; define a match-conditioned posterior and quantify its Rényi-2 entropy, which decomposes into a context term (log of answer-space size) and a multiplicity term (coefficient-of-variation of matching probabilities), estimated by an MLLM-as-verifier and lightweight Transformer heads trained on Cambrian-derived candidate answer set

Key results

• On Qwen3VL-2B, CoMet improves average open-ended VQA AUROC to .785 from .685 for the strongest baseline P(Correct), a +0.100 gain.
• CoMet outperforms VL Uncertainty by about +0.135 average AUROC on open-ended VQA while being generation-free.
• Ablation on Qwen3VL-8B: using both terms gives 0.798 average AUROC versus 0.780 for the multiplicity term alone and 0.686 for the context term alone.

Surprising findings

• A matching-probability module trained with contrastive loss on Cambrian learns to match based on the text question while ignoring the image, reaching near-random AUROC (0.51) when questions are mixed with random answers.

Limitations

• Hyperparameters lambda, beta, gamma are tuned on MMMU validation and a lightweight adaptation on benchmark-related data is needed for generalization.
• Candidate-answer approximation of p(y|t) from Cambrian clusters may not cover all plausible answers.
• Underperforms consistency-based baselines on some MCQ settings (e.g., InternVL3.5-1B).

Experimental setup

Datasets: Cambrian VQA v2 MMMU VizWiz OK-VQA HallusionBench MMMU Pro MMStar

Benchmarks: VQA v2 VizWiz OK-VQA HallusionBench MMMU MMMU Pro MMStar

Metrics: AUROC AUPR FPR@95

Baselines: Consistency entropy Consistency maximum probability NLL Verbalized confidence P(Correct) VL Uncertainty Candidate entropy Candidate maximum probability

Builds on: P(True) self-evaluation Cambrian dataset

Closest papers by meaning

• CertainlyUncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness (ICLR 2025 · similarity 0.89)
• FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty (ICML 2026 · similarity 0.89)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.88)
• Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.88)
• Localizing Input Uncertainty Quantification for Large Language Models via Shapley Values (NeurIPS 2026 · similarity 0.88)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.87)