Research map / Vision-language and multimodal / Multimodal LLMs

Contextualized Evaluation of Vision Language Models through Dynamic Interviews

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CEDI, a three-party (evaluatee, scene-graph-guided examiner, SG-based grader) dynamic multi-turn interview framework that elicits more visual hallucinations than static caption and binary-question evaluations.

Problem

Static, single-turn benchmarks for visual hallucination in MLLMs miss hallucinations that arise in dynamic, multi-turn, contextualized real-world use.

Approach

An examiner LLM conducts semi-structured multi-turn interviews by traversing a scene graph of the image under a user scenario and goal, choosing among regular, follow-up, adversarial, and unanswerable question types; a scene-graph-based grader (GED, SG-ΔCon) scores the transcript for hallucination and coverage.

Key results

• CEDI elicits more hallucinations than caption and prompt-only examiners on every evaluatee on SVG, with Coverage up 3-19 pp and VALOR per-turn hallucination rate up 5-23 pp.
• GED_hal achieves Pearson rho of 0.22-0.33 with human hallucination counts across four annotated evaluatees, while MMHal ranges -0.02 to 0.33 and CHAIR_I is negative for three of four models.
• Human annotation: CEDI sentence-level hallucination rate is 26.6% for InternVL3-8B versus 15.0% for caption, and 38.3% for LLaVA-1.5-7B versus 18.3% for caption.

Surprising findings

• Conversation history amplifies hallucination judgments but reduces coverage, as models repeat and extend earlier claims rather than probing new visual content.
• Adding the image to the examiner backbone does not improve hallucination elicitation and lowers several metrics.

Limitations

• Depends on ground-truth scene graphs to ground the examiner and score responses.
• Examiner and grader quality bounded by LLM/MLLM backbones (default GPT-4o).
• Transcript-level SG grader is unsuitable or costly for per-turn grading.

Experimental setup

Datasets: Visual Genome Synthetic Visual Genome COCO ADE20K

Benchmarks: Visual Genome Synthetic Visual Genome (COCO/VG/ADE20K)

Metrics: GED SG-ΔCon CHAIR_I CHAIR_S Coverage MMHal HaELM VALOR faith_i VALOR cov_i Pearson correlation with human hallucination counts

Baselines: Caption examiner POPE Prompt-only examiner CHAIR MMHal HaELM VALOR OPERA

Builds on: Scene Graphs CHAIR MMHal POPE SG-ΔCon

Closest papers by meaning

• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
• What matters when building vision-language models? (NeurIPS 2024 · similarity 0.89)
• Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.89)
• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.89)
• Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.89)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.89)