Research map / Vision-language and multimodal / Multimodal LLMs
Learning Interleaved Image-Text Comprehension in Vision-Language Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the IITC task and VEGA dataset for locating relevant text and images in long interleaved image-text contexts, and fine-tunes Qwen-VL-Chat with multi-task, multi-scale training to set a baseline.
Problem
Current MLLMs and benchmarks handle narrow visual-textual contexts and struggle to answer questions over long interleaved image-text documents containing distractors while identifying the referenced image.
Approach
Define Interleaved Image-Text Comprehension (IITC), which requires answering a question and outputting the index of the referenced image within long interleaved contexts, and pair it with an auxiliary Image-Text Association (ITA) task trained via multi-task, multi-scale post-training of Qwen-VL-Chat.
Key results
- VEGA-Base-4k (fine-tuned Qwen-VL-Chat) reaches 85.8% image association accuracy on IITC 4k and 0.508 Rouge-L, outperforming GPT4V (80.5%) and Gemini-1.5-pro (75.3%).
- On ITA with 5 images, Gemini-1.5-pro scores 0.891 and VEGA-Base-4k scores 0.992 accuracy.
- Removing multi-task and multi-scale training drops IITC 4k accuracy from 85.8% to 83.8%, and without multi-scale the ITA 5-image accuracy is 0.004.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Qwen-VL-Chat on VEGA instead of SciGraphQA | Qwen-VL-Chat · IITC 8k | Acc: 0.000 -> 0.775 | Helped |
| Train Qwen-VL-Chat on VEGA instead of SciGraphQA | Qwen-VL-Chat · SciGraphQA | Rouge-L: 0.538 -> 0.522 | Hurt |
| Use multi-scale ITA training rather than single-scale expanded context (ablation) | Qwen-VL-Chat with joint IITC+ITA training · ITA 3 pic | Image association accuracy: 0.155 -> 0.998 | Helped |
| Use multi-scale ITA training rather than single-scale expanded context (ablation) | Qwen-VL-Chat with joint IITC+ITA training · ITA 5 pic | Image association accuracy: 0.004 -> 0.992 | Helped |
| Add joint ITA training to IITC-only training (ablation) | Qwen-VL-Chat · IITC 4k | Acc: 0.838 -> 0.833 | Mixed |
| Add joint ITA training to IITC-only training (ablation) | Qwen-VL-Chat · IITC 8k | Acc: 0.766 -> 0.779 | Helped |
| Use multi-scale training with joint IITC+ITA training (ablation) | Qwen-VL-Chat · IITC 4k | Acc: 0.833 -> 0.858 | Helped |
| Use multi-scale training with joint IITC+ITA training (ablation) | Qwen-VL-Chat · IITC 8k | Acc: 0.779 -> 0.789 | Helped |
| Use the paper-expanded context construction instead of cross-paper merged context (ablation) | Qwen-VL-Chat trained on VEGA · IITC 8k | Acc: 0.521 -> 0.775 | Helped |
| Use the paper-expanded context construction instead of cross-paper merged context (ablation) | Qwen-VL-Chat trained on VEGA · SciGraphQA | Rouge-L: 0.507 -> 0.522 | Helped |
Surprising findings
- Training only on the IITC task without multi-scale ITA training yields ITA accuracy near random guessing (0.004 on 5 images).
- Context built by merging images and paragraphs from different papers (VEGA*) underperforms the same-paper expansion method.
Limitations
- Test set is small (about 700 questions) and restricted to scientific papers from arXiv.
- Proprietary model evaluations exclude unanswered instances, and Qwen-Max inputs were resized to 448x448 and truncated by API limits.
- Only one open-source model (Qwen-VL-Chat) is fine-tuned, so generality across architectures is untested.
Experimental setup
Datasets: SciGraphQA VEGA
Benchmarks: VEGA IITC 4k VEGA IITC 8k VEGA ITA SciGraphQA
Metrics: Rouge-L BLEU image association accuracy
Baselines: Gemini-1.5-pro Gemini-1.0-pro GPT4V Qwen-Max InternVL-1.5 Qwen-VL-Chat
Builds on: SciGraphQA Qwen-VL
Closest papers by meaning
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.91)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.90)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.89)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.89)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.