Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Auto-Comp, an automated pipeline generating photorealistic compositional binding benchmarks with paired Minimal/Contextual conditions and a Confusion hard-negative set, evaluating 25+ VLMs on color, shape-color, position, and relative size.

Problem

Existing compositional VLM benchmarks either use noisy real images that confound visual variables or simplistic synthetic scenes lacking realism, and swap-style hard negatives no longer discriminate strong modern MLLMs.

Approach

A concept-driven, fully automated pipeline that generates each concept in parallel as a template-caption white-background Minimal condition and an LLM-rewritten caption in a realistic scene (Contextual), validated by GroundedSAM2 and VLM checks, plus a Confusion hard-negative set built from low-entropy recombinations with replacement.

Key results

• Top contrastive models reach roughly 70-75% Swap on Color N=2 but near chance at N=3, and Confusion accuracy is consistently far below Swap across all benchmarks.
• Visio-linguistic context hurts attribute binding Confusion (e.g., Qwen2.5-VL-72B Color N=3 Confusion drops 46.8 to 31.9) but helps relational Swap (up to +17.4 points for SigLIP2-Giant on Position N=3).
• Hard-negative trained models (NegCLIP, NegCLIP++, TripletCLIP) improve Swap but leave Confusion within a few points of baseline; human accuracy is about 96% on both Swap and Confusion.

Surprising findings

• Confusion errors are not dominated by swapped colors: at N=2, over half of errors come from other low-entropy distractors.
• Contextual rendering reduces attribute-binding Confusion accuracy while improving relational Swap accuracy, a task-dependent trade-off.

Limitations

• Covers only currently renderable and verifiable compositional patterns; logical, temporal, causal and agent-object-action reasoning are not covered.
• Residual generator priors remain, notably in spatial relations and harder attribute combinations, with per-relation accuracy asymmetries that cannot be fully attributed to models versus data.
• Validation relies on a VLM judge (Gemma-3-12b-it) with about 94% human concordance, so judge errors can remain.

Experimental setup

Datasets: Object365 vocabulary

Benchmarks: Auto-Comp Color Auto-Comp Shape-Color Auto-Comp Position Auto-Comp Relative Size

Metrics: Swap accuracy Confusion accuracy

Baselines: CLIP SigLIP SigLIP2 NegCLIP NegCLIP++ TripletCLIP Qwen2.5-VL-72B InternVL3-78B

Builds on: CREPE SugarCrepe CIVET SPEC NegCLIP

Closest papers by meaning

• Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.92)
• Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.92)
• Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.91)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.90)
• Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.90)
• Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.90)