Research map / Vision-language and multimodal / Multimodal LLMs
Causal Graphical Models for Vision-Language Compositional Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COGT, a semi-parallel generative decoder over a causal graphical model built from dependency parses, trained on a frozen VLM encoder to improve compositional understanding.
Problem
Vision-language models treat captions as bags of words and perform poorly on compositional reasoning tasks requiring understanding of entities, relations and word order.
Approach
Builds a causal graphical model from a dependency tree so each word is predicted conditioned only on its syntactic ancestors and syntactic type (via type-specific masked tokens), with dependency-guided attention and level-order parallel inference for caption likelihood scoring.
Key results
- COGT-CLIP+ reaches 84.83 average accuracy across five compositional benchmarks, about 22 points above the second-best (DAC-LLM) as stated in the paper.
- COGT-XVLM+ reaches 86.30 average and COGT-InstructBLIP+ 85.87 average across the benchmarks.
- COGT-CLIP (COCO only) improves over the Sequential-AR (Cap-like) re-implementation from 69.28 to 75.20 average in the ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CGM-guided semi-parallel prediction instead of fully parallel prediction (ablation) | CLIP caption decoder · Compositional benchmarks (Avg) | accuracy: 57.42 -> 75.20 | Helped |
| Use CGM-guided semi-parallel prediction instead of mixed sequential-parallel prediction (ablation) | CLIP caption decoder · Compositional benchmarks (Avg) | accuracy: 67.10 -> 75.20 | Helped |
| Use CGM-guided semi-parallel prediction instead of sequential autoregressive prediction (ablation) | CLIP caption decoder · Compositional benchmarks (Avg) | accuracy: 69.28 -> 75.20 | Helped |
| Use Deep Biaffine + RoBERTa instead of CRFPar for parsing (ablation) | COGT · Compositional benchmarks (Avg) | accuracy: 71.76 -> 75.20 | Helped |
| Use Deep Biaffine + RoBERTa instead of Deep Biaffine for parsing (ablation) | COGT · Compositional benchmarks (Avg) | accuracy: 73.34 -> 75.20 | Helped |
| Add category-specific masked tokens instead of using a generic masked token (ablation) | COGT · Compositional benchmarks (Avg) | accuracy: -2.69 | Helped |
| Add penultimate-layer visual features alongside last-layer features (ablation) | COGT · Compositional benchmarks (Avg) | accuracy: -4.75 | Helped |
| Train on COCO, CC3M, and Visual Genome rather than COCO only (ablation) | COGT · Compositional benchmarks (Avg) | accuracy: about 9 points | Helped |
| Use COGT-CLIP rather than DAC-LLM | CLIP-based compositional methods · Compositional benchmarks (Avg) | accuracy: 12.27 points | Helped |
| Use COGT rather than zero-shot InstructBLIP | InstructBLIP · Winoground | Group Score: +7.5 points | Helped |
| Use COGT-InstructBLIP+ rather than InstructBLIP-VQA Score | InstructBLIP · Winoground | Group Score | Hurt |
| Use COGT-CLIP rather than CLIP | CLIP visual features with linear probing · ImageNet1K top 1 | accuracy | No clear effect |
Surprising findings
- A standard fully-parallel decoder performs worse than the mixed strategy by a wide margin, indicating unconstrained conditional independence is too strong.
- COGT-CLIP improves ImageNet-related linear probing over CLIP-based baselines and does not degrade standard visual features.
Limitations
- Relies on an off-the-shelf dependency parser whose errors (especially on noisy CC3M captions) propagate; better parsers were not used.
- Assumes dependency relations are causal and causally sufficient, which is a heuristic assumption not formally verified.
- Underperforms LLM-based or VQA-score methods on Winoground and does not beat InstructBLIP-VQA Score there.
Experimental setup
Datasets: COCO CC3M Visual Genome ARO SugarCrepe VL-CheckList ColorSwap FG-OVD
Benchmarks: ARO SugarCrepe VL-CheckList ColorSwap FG-OVD Winoground MMVP CIFAR10 CIFAR100 ImageNet1K
Metrics: accuracy group score
Baselines: CLIP NegCLIP CE-CLIP Plausible Adj. Neg DAC-SAM DAC-LLM CE-XVLM Cap CapPa BLIP-VisualGPTScore Sequential-AR Fully-Parallel
Builds on: Cap CapPa NegCLIP CLIP Deep Biaffine parser
Closest papers by meaning
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.90)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.90)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.90)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.89)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.