Research map / Vision-language and multimodal / Visual question answering
EDMA: Entropy-Driven Multimodal Answering
Emanuele Mezzi, Gertjan Burghouts, Fabio Massacci, Mengyuan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDMA, an entropy-driven approach to multimodal question answering that quantifies how much each modality contributes to the final answer.
Problem
Multimodal question answering must integrate heterogeneous sources, select relevant modalities, and ignore distractors; existing MLLM-based methods split the task into intermediate steps but do not quantify each modality's contribution to the final answer.
Approach
Quantify the effect of each modality on reaching the final answer using entropy, and use this entropy signal to drive modality selection and answering, rather than relying on unquantified intermediate steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EDMA entropy-driven answering instead of Planning Prompting | 6 MLLMs · MultiModalQA (val) | EM / F1 / LLM-J: +22.7 EM, +14.0 F1, +8.5 LLM-J | Improved |
| use EDMA instead of Planning Prompting | 6 MLLMs · ManyModalQA | EM / F1 / LLM-J: +23.0 EM, +11.5 F1, +0.6 LLM-J | Improved |
| select trajectory by minimum logical entropy instead of uniformly at random (ablation) | EDMA · MultiModalQA (val), 6 MLLMs | LLM-J: 47.2 -> 79.3 | Improved |
| add cross-modal conditioning (multimodal partitions) on top of unimodal partitions (ablation) | EDMA · MultiModalQA (val) | LLM-J: 65.5 -> 80.5 (GPT-5.2), 48.0 -> 65.9 (Nova-2-Lite) | Improved |
| use EDMA instead of self-consistency (N=6) baselines at matched cost | CoT-SC and PP-SC · MultiModalQA (val), 4 MLLMs | LLM-J: 71.2 / 70.8 -> 78.2 | Improved |
| use EDMA instead of multi-agent MAMMQA | MAMMQA · MultiModalQA (val), 4 MLLMs | LLM-J: 75.7 -> 78.2 average; Kimi-K2.5 88.4 vs 83.2 | Mixed |
| apply self-consistency (N=6) to Planning Prompting on larger backbones (ablation) | PP · MultiModalQA (val), Claude-4.6 | EM: -4.9 EM, -3.2 F1, -3.1 LLM-J | Hurt |
| apply self-consistency (N=6) to Planning Prompting on Nova-2-Lite (ablation) | PP · MultiModalQA (val), Nova-2-Lite | LLM-J: +5.5 EM, +9.5 F1, +17.8 LLM-J | Improved |
| choose lowest-entropy trajectory instead of median or highest entropy (ablation) | EDMA · MultiModalQA, all target modalities | false discovery rate | Improved |
| apply EDMA to cross-dataset ManyModalQA versus prior specialized systems | UniMMQA · ManyModalQA | EM: 52.1 -> 88.0 | Improved |
| use EDMA instead of single-pass prompting in compute terms | EDMA · compute cost | cost ratio vs CoT: 5.91x cost, 22.83x latency vs CoT | Hurt |
Closest papers by meaning
- Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
- Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution (AAAI 2025 · similarity 0.87)
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.86)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.86)
- Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models (CVPR 2024 · similarity 0.86)
- Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
