Research map / Vision-language and multimodal / Multimodal LLMs
Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multimodal Composition Learning (MCL), training a frozen LLM with a LLM-generated multimodal composition dataset (MMC, 2.7M tuples) and stacked retrieval tokens to improve multimodal context understanding and retrieval.
Problem
Frozen-LLM vision-language mappings trained only on image captioning and image-text retrieval handle complex multimodal contexts (composed image retrieval, multi-turn dialog, storytelling) poorly, and paired multimodal composition data is scarce due to manual labeling.
Approach
Augment web image-caption pairs with an LLM (Llama2) into (ref image, text condition, target caption) tuples, then train a linear visual adaptor and learnable [RET] tokens with MC-Cap and MC-Ret objectives on top of a frozen LLM, using a stacking retrieval mechanism with attention masks between [RET] tokens to extract diverse information.
Key results
- MCL with Llama2-7B reaches 17.67 mAP@5 on CIRCO, outperforming CompoDiff (12.55) and FROMAGe (4.00 with OPT-6.7B).
- On Visual Dialog image retrieval, MCL (Llama2-7B) achieves R@1 29.8 vs. 20.8 for FROMAGe (OPT-6.7B) and 17.7 for CLIP ViT-L/14.
- On VQAv2 zero-shot, MCL (Llama2-7B) reaches 42.6 accuracy, versus 36.8 for FROMAGe and 38.8 for GILL (OPT-6.7B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with multimodal composition learning objectives (MC-Cap and MC-Ret) instead of naive mapping objectives (ablation) | MCL · CIRCO test | mAP@5: 4.38 -> 17.55 | Helped |
| add MC-Cap to naive mapping objectives (ablation) | MCL · CIRCO test | mAP@5: 4.38 -> 6.58 | Helped |
| add MC-Ret to naive mapping objectives (ablation) | MCL · CIRCO test | mAP@5: 4.38 -> 16.73 | Helped |
| use stacking retrieval rather than sequential ordering for five [RET] tokens (ablation) | MCL · CIRCO test | mAP@5: 16.48 -> 17.67 | Helped |
| increase retrieval tokens from one to five without stacking retrieval (ablation) | MCL · CIRCO test | mAP@5: 17.07 -> 16.48 | Hurt |
| add naive image captioning and image-text retrieval objectives to MC-Cap and MC-Ret (ablation) | MCL · CIRCO test | mAP@5: 17.55 -> 17.67 | Helped |
| use MCL rather than ComposDiff | zero-shot CIR · CIRCO | mAP@5: 12.55 -> 17.67 | Helped |
| use MCL rather than FROMAGe | visual dialogue image retrieval · Visual Dialog | R@1: 20.8 -> 29.8 | Helped |
| increase input context from one caption to five captions | MCL (Llama2-7B) · Visual Storytelling | R@1: 11.4 -> 13.7 | Helped |
| increase input context from one caption to five captions | CLIP ViT-L/14 · Visual Storytelling | R@1: 11.9 -> 5.9 | Hurt |
| use MCL with Llama2-7B rather than MCL with OPT-6.7B | MCL · CIRCO | mAP@5: 15.14 -> 17.67 | Helped |
Surprising findings
- Using multiple [RET] tokens without stacking performs worse than a single [RET] token, since adjacent retrieval tokens attend to similar content.
Limitations
- Zero-shot VQAv2 results lag behind recent SOTA MLLMs trained with more compute and in-domain data.
- Multiple [RET] tokens without stacking degrade performance due to adjacent-token interference.
Experimental setup
Datasets: MMC (generated from web image-caption pairs) CIRCO CIRR GeneCIS Visual Storytelling Visual Dialog VQAv2 MSCOCO
Benchmarks: CIRCO CIRR GeneCIS Visual Storytelling Visual Dialog VQAv2
Metrics: mAP@K Recall@K accuracy
Baselines: Image-only Text-only Image+Text Pic2Word SEARLE CompoDiff Combiner-MMC FROMAGe GILL CLIP ViT-L/14
Builds on: FROMAGe GILL Combiner CLIP
Closest papers by meaning
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.91)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.90)
- Towards More Unified In-context Visual Understanding (CVPR 2024 · similarity 0.89)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.89)
- FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.89)
- Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.