Research map / Vision-language and multimodal / Multimodal LLMs
Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromViL, which fine-tunes grounded LVLMs (Kosmos-2, MiniGPTv2) with LoRA on a new CompoVL dataset of nested compositional vision-language pairs built from Visual Genome, and prompts them progressively from simple to complex expressions.
Problem
Large VLMs struggle with compositional visual grounding and relational expressions involving multiple similar objects, as their grounding is coarse (whole phrase to region) or fine (single-object phrases) and lacks intermediate relational structure.
Approach
Decompose compositional expressions into nested subsequences of increasing complexity, build a nested V-L dataset (CompoVL) from Visual Genome predicates with an LLM, and train the LVLM to ground each level conditioned on the previous level's expression and predicted box, then decode progressively from simple to complex.
Key results
- On zero-shot RefCOCOg test, PromViL (1.6B) reaches 65.28 vs Kosmos-2 61.65; on CompoVL-hard it reaches 64.07 vs 55.37 for Kosmos-2.
- With MiniGPTv2 backbone (7B), PromViL reaches 68.92 on CompoVL-hard vs 62.57 for MiniGPTv2 and 85.32 on RefCOCOg test vs 84.66.
- On GQA val, PromViL achieves 45.07 accuracy vs 41.44 for Kosmos-16K; on Visual7W PromViL reaches 50.28 test accuracy vs 47.96 for Kosmos-16K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use progressive multi-granular alignment instead of standard Kosmos-2 | Kosmos-2 · CompoVL-hard | accuracy: 55.37 -> 64.07 | Helped |
| use progressive multi-granular alignment instead of standard Kosmos-2 | Kosmos-2 · RefCOCOg test | accuracy: 61.65 -> 65.28 | Helped |
| use progressive multi-granular alignment instead of Kosmos-16K | Kosmos-2 · GQA val | accuracy: 41.44 -> 45.07 | Helped |
| use progressive multi-granular alignment instead of Kosmos-16K | Kosmos-2 · GQA val | validity: 75.14 -> 83.47 | Helped |
| use progressive multi-granular alignment instead of Kosmos-16K | Kosmos-2 · Visual7W test | accuracy: 47.96 -> 50.28 | Helped |
| add intermediate expression levels to the training chain (ablation) | PromViL · CompoVL-hard | accuracy: 61.22 -> 64.07 | Helped |
| add intermediate expression levels to the training chain (ablation) | PromViL · RefCOCOg test | accuracy: 63.44 -> 65.28 | Helped |
| add lower and intermediate levels alongside highest-level expressions (ablation) | PromViL · CompoVL-hard | accuracy: 59.12 -> 64.07 | Helped |
| add lower and intermediate levels alongside highest-level expressions (ablation) | PromViL · RefCOCOg test | accuracy: 62.69 -> 65.28 | Helped |
| add higher and intermediate levels alongside simplest-level expressions (ablation) | PromViL · CompoVL-hard | accuracy: 53.26 -> 64.07 | Helped |
| add higher and intermediate levels alongside simplest-level expressions (ablation) | PromViL · RefCOCOg test | accuracy: 59.92 -> 65.28 | Helped |
Surprising findings
- Fine-tuning Kosmos-2 on CompoVL without nested subsequences yields only marginal gains (55.37 to 59.12 on CompoVL-hard), suggesting the progressive structure drives most of the improvement.
- Doubling the VQA training data (Kosmos-8K to Kosmos-16K) yields minimal improvement on GQA and Visual7W.
Limitations
- CompoVL dataset release is not explicitly stated; reliance on LLM-generated expressions (Mixtral) with possible hallucinations, mitigated only by prompting and a 2% human check.
- Inference relies on constituency and dependency parsers to decompose expressions, which may introduce errors.
- Gains over Kosmos-2 on some tasks (e.g., GQA open-ended accuracy, GQA distribution) are small or mixed.
Experimental setup
Datasets: CompoVL Visual Genome VSR VG-VQA LLaVA-Instruct-150K RefCOCOg RefCOCO RefCOCO+ GQA GQA-OOD Visual7W
Benchmarks: CompoVL-hard RefCOCOg RefCOCO RefCOCO+ GQA GQA-OOD Visual7W
Metrics: top-1 accuracy GQA accuracy/binary/open/consistency/validity/plausibility/d GQA-OOD acc_all/acc_tail/acc_head
Baselines: Kosmos-2 Pink MiniGPTv2 CoVLM F-Kosmos-2 Kosmos-8K Kosmos-16K
Builds on: Kosmos-2 MiniGPTv2 LoRA Visual Genome
Closest papers by meaning
- Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.91)
- Beyond the Linear Separability Ceiling: Aligning Representations in VLMs (NeurIPS 2026 · similarity 0.90)
- Vision-aligned Latent Reasoning for Multi-modal Large Language Model (ICML 2026 · similarity 0.90)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.90)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.