Research map / Vision-language and multimodal / Multimodal LLMs

Multimodal Fact-Level Attribution for Verifiable Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MuRGAt, a benchmark and automatic fact-level evaluation pipeline (MuRGAt-Score) for multimodal reasoning answers with timestamped, per-modality citations, showing strong MLLMs often hallucinate citations despite correct answers.

Problem

Existing multimodal grounding benchmarks focus on observation-based, visual-only settings and do not assess fact-level attribution for multi-step reasoning over heterogeneous inputs such as video, audio, and figures.

Approach

Decomposes response evaluation into verifiable claim identification, atomic fact decomposition, and set-based attribution (recall and precision over timestamped, modality-specific citations), combined into a single score (Coverage x Attribution) and calibrated by choosing the best LLM per subtask against human annotations.

Key results

• The automatic MuRGAt-Score correlates with human judgments at Pearson 0.86 (Spearman 0.84) in the final configuration, versus 0.59 for the best LLM-as-judge baseline reported in the introduction.
• Gemini-3-Flash with citation reaches the peak MuRGAt-Score of 69.2 on WorldSense and 56.9 on Video-MMMU, with attribution as the main bottleneck.
• Programmatic imperative grounding improves MuRGAt-Score by an average of 9.6 points over Base+Citation on WorldSense but lowers answer accuracy by about 7.4 points on average.

Surprising findings

• Higher answer accuracy does not imply better grounding: Gemini-3-Pro matches Gemini-3-Flash accuracy on Video-MMMU (86.0) yet has much lower MuRGAt-S (41.8 vs 56.9).
• Citation generation acts as a reasoning tax on WorldSense but improves accuracy on Video-MMMU, e.g., Qwen3-VL-Thinking rises from 51.0 to 60.0.

Limitations

• Human evaluation covers only 20 inputs and 80 responses from four models, so correlations rest on a small sample.
• Evaluation pipeline depends on proprietary Gemini models as judges, and Qwen evaluators achieve substantially lower correlation.
• Models tend to dump citations at sentence end, and the blanket citation propagation to atomic facts is an approximation.

Experimental setup

Datasets: WorldSense Video-MMMU

Benchmarks: MuRGAt (WorldSense) MuRGAt (Video-MMMU)

Metrics: Coverage Attribution F1 MuRGAt-Score Accuracy Balanced Accuracy Pearson correlation Spearman correlation ROUGE-1 F1

Baselines: Holistic LLM-as-judge Disentangled LLM-as-judge Disentangled (sentence-level) LLM-as-judge Gemini-2.5-Flash Gemini-3-Pro Qwen3-Omni-Instruct Qwen3-Omni-Thinking

Builds on: program-aided generation ViperGPT FActScore-style atomic fact evaluation decontextualization

Closest papers by meaning

• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.89)
• Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark (ICML 2025 · similarity 0.88)
• MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.88)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.88)
• GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking (CVPR 2026 · similarity 0.88)