Research map / Vision-language and multimodal / Visual question answering
DIEM: Decomposition-Integration Enhancing Multimodal Insights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free decomposition-integration method that splits questions into subquestions (GPT-3.5-Turbo), segments images into sub-images (SAM), matches them via CLIP, and answers with LLaVA-1.5-7b, improving ScienceQA and MM-Vet scores.
Problem
Precisely matching and integrating textual and visual information in image question answering, where images contain abundant and sometimes redundant details that MLLMs may overlook or misfocus on.
Approach
Decompose the question into subquestions and the image into sub-images, match each subquestion to its top-k relevant sub-images while always keeping the full image, then answer subquestions sequentially and integrate subquestion-subanswer pairs to answer the original question, without any training.
Key results
- On ScienceQA, DIEM (SQ+SI, k=5) raises average accuracy from 66.85% to 68.88% (+2.03) and image-context accuracy by +3.40 points to 69.10%.
- On MM-Vet, DIEM improves total score from 31.1 to 32.4 (k=4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decompose questions into subquestions without using sub-images (ablation) | LLaVA-1.5-7B · ScienceQA test | average accuracy (%): 66.85 -> 68.64 | Helped |
| add sub-image decomposition and matching to question decomposition (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA test | average accuracy (%): 68.64 -> 68.88 | Helped |
| decompose questions into subquestions without using sub-images (ablation) | LLaVA-1.5-7B · ScienceQA image modality | accuracy (%): 65.70 -> 68.64 | Helped |
| add sub-image decomposition and matching to question decomposition (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA image modality | accuracy (%): 68.64 -> 69.10 | Helped |
| decompose questions into subquestions without using sub-images (ablation) | LLaVA-1.5-7B · MM-Vet | Total score: 31.1 -> 31.3 | Helped |
| add sub-image decomposition and matching to question decomposition (ablation) | DIEM on LLaVA-1.5-7B · MM-Vet | Total score: 31.3 -> 32.4 | Helped |
| omit the original question during integration (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA image modality | accuracy (%): 69.01 -> 66.61 | Hurt |
| omit the original image during integration (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA image modality | accuracy (%): 69.01 -> 68.50 | Hurt |
| omit the original image during integration (ablation) | DIEM on LLaVA-1.5-7B · MM-Vet | Total score: 32.4 -> 31.3 | Hurt |
| provide context, but no options, to subquestions (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA test | average accuracy (%): 68.29 -> 68.83 | Helped |
| also provide options to subquestions (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA test | average accuracy (%): 68.83 -> 68.19 | Hurt |
| provide context, but no options, to subquestions (ablation) | DIEM on LLaVA-1.5-7B · ScienceQA image modality | accuracy (%): 68.50 -> 69.01 | Helped |
Surprising findings
- Providing answer options to subquestions drops ScienceQA accuracy by 0.64%, as biased subanswers mislead the final answer.
- Using all sub-images (k=ALL) causes accuracy to drop significantly, attributed to noise from images with few objects.
Limitations
- Relies on GPT-3.5-Turbo with 4k context and few-shot prompts for subquestion decomposition, limiting decomposition quality.
- Decomposition can lose textual information and sever inter-element relations in diagrams such as flowcharts.
- Gains are small (about 2 points) and degrade when too many sub-images are used or images are simple.
Experimental setup
Datasets: ScienceQA MM-Vet
Benchmarks: ScienceQA MM-Vet
Metrics: accuracy MM-Vet total score
Baselines: LLaVA-1.5-7b standard prompting Multimodal-CoT
Builds on: chain-of-thought LLaVA-1.5 Segment Anything CLIP
Closest papers by meaning
- Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (AAAI 2025 · similarity 0.88)
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.87)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.87)
- MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding (ICLR 2025 · similarity 0.86)
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.86)
- DEQA: Descriptions Enhanced Question-Answering Framework for Multimodal Aspect-Based Sentiment Analysis (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.