Research map / Vision-language and multimodal / Visual question answering

SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring

NeurIPS 2026 · Vision-language and multimodal › Visual question answering · approach: Multimodal LLM question answering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SIEVES, a model-agnostic selective prediction method for VQA that has reasoners zoom in with crops and trains a small Gemma-3-4B selector to score answer correctness, localization, and coherence from observable outputs, transferring zero-shot to o3 and Gemini-3-Pro.

Problem

Selective prediction for multimodal LLMs in out-of-distribution VQA: existing confidence estimators rely on model internals (logits, hidden states) unavailable for proprietary frontier models, and verification-based methods are costly.

Approach

Require the reasoner to ground its answer with zoom-in crops (visual evidence) and train a small selector on observable outputs (question, image, MM-CoT, answer) with three heads predicting correctness, localization quality (IoGT-based binary label), and crop-answer coherence (VLM-judged label), combined into a single confidence score.

Key results

• On V* Bench with Pixel-Reasoner, SIEVES reaches C@5 of 9.7 vs 2.3 for the correctness-only selector and 0.2 for logprobs baseline (the abstract claims up to 3x coverage gains).
• Transferred without retraining to o3 on V* Bench, SIEVES achieves C@5 of 23.1 vs 0.1 for the correctness-only selector.
• With Gemini-3-Pro on V* Bench, SIEVES achieves C@5 of 95.0 vs 8.8 for the correctness-only selector.

Surprising findings

• A correctness-only selector trained on localized (tool-augmented) answers sometimes performs worse than one trained on non-localized answers, e.g. on MME-RealWorld-Lite and V* Bench.
• Zero-shot verbalized and logit selectors perform very poorly, often achieving near-zero coverage at low risk.

Limitations

• Relies on the reasoner producing zoom-in crops; prompts force at least one zoom-in, altering reasoner behavior.
• Localization and coherence labels depend on bounding boxes (human-annotated for training) and a VLM judge; human check of coherence labels covered only 200 samples (92% agreement).
• Ground-truth localization analysis only possible on V* Bench, the only OOD benchmark with bounding boxes.

Experimental setup

Datasets: Thyme TAT-DQA V* Bench HR-Bench-8k MME-RealWorld-Lite VizWiz AdVQA

Benchmarks: V* Bench HR-Bench-8k MME-RealWorld-Lite VizWiz AdVQA Thyme holdout

Metrics: coverage at risk (C@r) average coverage AURC accuracy

Baselines: Logprobs Zero-shot verbalized Zero-shot logit Correctness-only selector

Builds on: Pixel-Reasoner LoRA LLaMA-Factory

Closest papers by meaning

• Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (CVPR 2024 · similarity 0.89)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
• Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.88)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.88)
• EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.88)
• When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.88)