Research map / Vision-language and multimodal / Multimodal LLMs

Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a verb-hallucination benchmark built from HICO and CharadesEgo for MLLMs, shows severe verb hallucination and failure of existing mitigation methods, and proposes LoRA fine-tuning of LLaVA-V1.5 on a verb-rich Pangea-derived instruction set.

Problem

MLLMs hallucinate about verb/action concepts, a category largely ignored by object-focused hallucination benchmarks and mitigation methods.

Approach

Builds verb-focused YN and MC probes by altering verbs while holding objects fixed, evaluates many MLLMs under varied conditions (object reference, image distortion, egocentric view, rare verbs, ambiguity), and mitigates via LoRA fine-tuning on 60K verb-rich Pangea-derived samples.

Key results

• Fine-tuning LLaVA V1.5 with LoRA on Pangea-derived verb data raises YN w/ obj accuracy from 52.16 to 78.48 and MC w/o obj accuracy from 51.00 to 60.79.
• Training-free mitigation methods (OPERA, VCD, Nullu) show marginal, inconsistent, or negative effects on verb hallucination.
• Visual pepper-salt noise degrades verb understanding more than object understanding, with lower error consistency for verb questions.

Surprising findings

• Training-free methods OPERA and VCD fail to reduce verb hallucination, and VCD's KL divergence is zero on most samples, suggesting heavy reliance on language prior.
• LLaVA V1.5 achieves higher mAP (68.41) than HICO-finetuned CLIP (60.45) despite severe verb hallucination in YN/MC accuracy, indicating miscalibration rather than lack of verb knowledge.

Limitations

• Fine-tuned model still far from satisfactory on verb hallucination.
• Transitive verbs mean object influence cannot be fully removed from benchmarks.
• Pangea labels are rough action labels, limiting the verb supervision quality.

Experimental setup

Datasets: HICO CharadesEgo Pangea POPE

Benchmarks: HICO-based verb benchmark CharadesEgo-based verb benchmark MMMU MathVista

Metrics: accuracy precision recall F1 mAP Cohen's kappa error consistency

Baselines: OPERA VCD Nullu REVERIE Haloquest Octopus LLaVA V1.5

Builds on: POPE OPERA VCD Nullu Pangea

Closest papers by meaning

• In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.90)
• ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models (NeurIPS 2024 · similarity 0.89)
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.89)
• Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.89)
• The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.89)
• Hallucination Augmented Contrastive Learning for Multimodal Large Language Model (CVPR 2024 · similarity 0.89)