Research map / Vision-language and multimodal / Visual question answering

CertainlyUncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness

ICLR 2025 · Vision-language and multimodal › Visual question answering · approach: Vision-language model VQA evaluation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a taxonomy of epistemic and aleatoric uncertainty for vision-language models, a 178K-sample VQA benchmark (CertainlyUncertain) built from captions and inpainted images, and a confidence-weighted accuracy metric; SFT on the data improves IDK behavior and calibration.

Problem

Vision-language models rarely admit uncertainty or refuse unanswerable questions, and existing metrics (accuracy, F1, ECE, risk/coverage) do not jointly capture correctness and confidence in such settings.

Approach

A taxonomy splitting multimodal uncertainty into epistemic (knowledge, complexity, extraneous) and aleatoric (temporal, ambiguity) categories, with contrastive answerable/unanswerable VQA pairs created by GPT-4 from captions and by inpainting images with Grounded-SAM and LaMa, plus a confidence-weighted accuracy metric that rewards confident correct answers and penalizes confident wrong ones.

Key results

• Fine-tuning Qwen-VL-Chat with LoRA-SFT on the dataset raises total confidence-weighted accuracy from 13.45 to 49.35 and reduces IDK ECE from 0.79 to 0.31.
• On refusal benchmarks, LoRA-SFT with the data only raises UNK-VQA accuracy from 41.32 to 60.65 and TDIUC from 95.10 to 99.64.
• Even GPT-4V scores only 71.70 LAVE_idk accuracy overall and models answer confidently on unanswerable questions, with LLaVA-1.6-34B reaching the best total confidence-weighted accuracy of 32.47 among evaluated open models.

Measured effects reported in the paper

Surprising findings

• GPT-4V fails to answer the uncertain questions it generated itself, mirroring the generative AI paradox.
• SFT on the new data degraded AMBER F1 for Qwen-VL-Chat, which the authors attribute to missing non-existent-object attribute and relation questions.

Limitations

• Synthetic pipeline may not capture the nuances of real-world uncertain scenarios.
• Best-performing SFT with LoRA is more resource-intensive than inference-time selective prediction.
• Dataset does not cover the absurd category and offers no reasonable-guess responses; the authors leave guessing to future work.

Experimental setup

Datasets: CertainlyUncertain DOCCI VQAv2 GQA LLaVA instruction data LRV instruction data MMInstruction UNK-VQA TDIUC MM-Hal POPE AMBER

Benchmarks: CertainlyUncertain UNK-VQA TDIUC MM-Hal POPE AMBER VizWiz VQAv2

Metrics: LAVE_idk accuracy F1_idk confidence-weighted accuracy ECE hallucination rate VQA score

Baselines: LLaVA-1.5-7B LLaVA-1.5-13B LLaVA-1.6-7B LLaVA-1.6-13B LLaVA-1.6-34B Qwen-VL-Chat GPT-4V LRV MMInstruction Thresholding

Builds on: LAVE R-tuning DPO LLaVA Grounded-SAM

Closest papers by meaning

• FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty (ICML 2026 · similarity 0.91)
• Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding (NeurIPS 2025 · similarity 0.89)
• CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation (NeurIPS 2026 · similarity 0.89)
• Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• Refine and Align: Confidence Calibration Through Multi-Agent Interaction in VQA (AAAI 2026 · similarity 0.88)
• NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.88)