Research map / Vision-language and multimodal / Multimodal LLMs

MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MoHoBench, a 12k-question benchmark of unanswerable visual questions across four types, to measure honesty (refusal) in 28 MLLMs, and tests SFT/DPO/SimPO/ORPO alignment for honest refusal.

Problem

MLLMs often fabricate answers to visually unanswerable questions instead of refusing; honesty of multimodal models is underexplored, and existing hallucination benchmarks do not measure refusal behavior.

Approach

Defines four categories of unanswerable visual questions (context dependent, false premises, subjective/philosophical, vague description), generates candidates with ICL from strong MLLMs, keeps hard cases that most strong models fail to refuse, and evaluates honesty via refusal rate plus LLM-judged rationality and helpfulness.

Key results

• Average refusal rate across 28 evaluated MLLMs is only 21.3%, and average refusal rationality is 6.09 out of 10.
• Llama-3.2-90B-Vision-Instruct has the highest refusal rate (55.3%) while QVQ-72B-Preview has the lowest (7.4%); model size correlates only weakly with refusal rate (Pearson 0.46, R^2 0.21).
• SFT, DPO, SimPO and ORPO raise honesty on Qwen2.5-VL-7B and InternVL2.5 models from roughly 13-29% to 83-99%, at a cost to rationality and sometimes MMMU.

Surprising findings

• Contrast reduction produced a rise in refusal for some models, and additive noise generally lowered refusal rates, suggesting over-confidence on degraded images.
• Larger models do not reliably refuse more; the 90B Llama model had the highest refusal yet the lowest rationality and helpfulness.

Limitations

• Taxonomy of four unanswerable categories may not cover all real-world unanswerability.
• Alignment methods (SFT, DPO, SimPO, ORPO) are existing techniques; no honesty-specific objective is proposed.
• Evaluation relies on LLM judges (o3-mini, GPT-4o) validated on a small human sample of 105 items.

Experimental setup

Datasets: COCO HaloQuest RLHF-V

Benchmarks: MoHoBench MMMU

Metrics: refusal rate refusal rationality general helpfulness balanced performance index

Baselines: SFT DPO SimPO ORPO GPT-4o o1

Builds on: DPO SimPO ORPO RLHF-V HaloQuest

Closest papers by meaning

• MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
• HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.90)
• Alignment for Honesty (NeurIPS 2024 · similarity 0.89)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
• MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.89)
• UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.88)