Research map / Vision-language and multimodal / Visual question answering
Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies Between Model Predictions and Human Responses in VQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates how well VQA 2.0 model confidence (BEiT3, LXMERT) aligns with human answer-confidence distributions across low/medium/high human uncertainty splits, using TVD, KL, and EntCE, and compares accuracy-oriented versus human-oriented temperature scaling.
Problem
Vision-language models are evaluated mainly by VQA-Accuracy on a single prediction, ignoring human disagreement and human uncertainty in responses, so their confidence may not match human answer distributions.
Approach
Quantify per-sample human uncertainty in disagreement (HUD) from VQA 2.0 yes/no/maybe confidence labels, split samples into low/medium/high HUD sets, evaluate models with distribution-based human-correlated metrics, and calibrate temperature scaling towards human distributions rather than accuracy.
Key results
- BEiT3 reaches VQA-Accuracy 0.842 overall and 0.759 on the high-HUD set, and outperforms LXMERT on TVD, KL, and EntCE in every split before calibration.
- Temperature scaling towards VQA-Accuracy worsens BEiT3's human alignment (KL rises from 1.701 to 3.232), while calibrating towards human distributions at T=2.0 reduces BEiT3's KL to 0.647 and TVD to 0.340.
- Both models score best on the high-HUD set on TVD, KL, and EntCE, and worst on the low-HUD set, despite accuracy being highest on the low set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase HUD level from low to high | BEiT3 and LXMERT · VQA 2.0 validation HUD subsets | VQA-Acc: BEiT3: 0.921 -> 0.759; LXMERT: 0.761 -> 0.576 | Hurt |
| Increase HUD level from low to high | BEiT3 and LXMERT · VQA 2.0 validation HUD subsets | TVD: BEiT3: 0.455 -> 0.416; LXMERT: 0.454 -> 0.439 | Helped |
| Increase HUD level from low to high | BEiT3 and LXMERT · VQA 2.0 validation HUD subsets | KL(Human||Model): BEiT3: 2.150 -> 1.408; LXMERT: 2.237 -> 1.634 | Helped |
| Increase HUD level from low to high | BEiT3 and LXMERT · VQA 2.0 validation HUD subsets | EntCE: BEiT3: 0.575 -> 0.503; LXMERT: 0.580 -> 0.506 | Helped |
| Calibrate toward VQA-Acc with temperature scaling (ablation) | LXMERT · VQA 2.0 validation, all | TVD: 0.448 -> 0.442 | Helped |
| Calibrate toward VQA-Acc with temperature scaling (ablation) | LXMERT · VQA 2.0 validation, all | KL: 1.886 -> 1.771 | Helped |
| Calibrate toward VQA-Acc with temperature scaling (ablation) | LXMERT · VQA 2.0 validation, all | EntCE: 0.555 -> 0.542 | Helped |
| Calibrate toward VQA-Acc with temperature scaling (ablation) | BEiT3 · VQA 2.0 validation, all | TVD: 0.436 -> 0.488 (-) | Hurt |
| Calibrate toward VQA-Acc with temperature scaling (ablation) | BEiT3 · VQA 2.0 validation, all | KL: 1.701 -> 3.232 (-) | Hurt |
| Calibrate toward VQA-Acc with temperature scaling (ablation) | BEiT3 · VQA 2.0 validation, all | EntCE: 0.541 -> 0.670 (-) | Hurt |
| Calibrate toward human distributions with temperature scaling (ablation) | LXMERT · VQA 2.0 validation, all | TVD: 0.448 -> 0.354 | Helped |
| Calibrate toward human distributions with temperature scaling (ablation) | BEiT3 · VQA 2.0 validation, all | TVD: 0.436 -> 0.340 | Helped |
Surprising findings
- Calibrating BEiT3 towards VQA-Accuracy with temperature scaling degrades its alignment with human distributions across all metrics.
- Both models align best with human distributions on the high-uncertainty set even though their VQA-Accuracy is lowest there.
Limitations
- Restricted to two slightly different validation splits and a single dataset (VQA 2.0).
- Uses a single standard scheme for mapping yes/maybe/no confidence labels to scores, which may not capture all uncertainty quantification options.
- Only temperature scaling is explored as a calibration method.
Experimental setup
Datasets: VQA 2.0
Benchmarks: VQA 2.0
Metrics: VQA-Accuracy TVD KL divergence EntCE ECE
Baselines: LXMERT BEiT3
Builds on: Temperature Scaling VQA 2.0 VQA-Accuracy
Closest papers by meaning
- Refine and Align: Confidence Calibration Through Multi-Agent Interaction in VQA (AAAI 2026 · similarity 0.89)
- Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (CVPR 2024 · similarity 0.89)
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.89)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.88)
- Right this way: Can VLMs Guide Us to See More to Answer Questions? (NeurIPS 2024 · similarity 0.88)
- Improving Automatic VQA Evaluation Using Large Language Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.