Research map / Vision-language and multimodal / Multimodal LLMs

A Causal Marriage between VLM and IRM from Understanding to Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Links CLIP's contrastive objective to IRM via token-level causal representation, proposes a vocabulary-constrained mid-training of CLIP (CLIP-IRM), and uses its invariant alignment scores as process rewards in GRPO for Qwen2.5-VL-7B reasoning.

Problem

Lack of theoretical explanation for CLIP's strong OOD generalization and no formal link between VLM contrastive training and invariant risk minimization; also how to transfer IRM-style invariance to MLLM reasoning.

Approach

Decomposes text prompts into class tokens (causal factors) and context tokens (environment factors), proves a vocabulary-constrained InfoNCE objective is equivalent to IRM's invariance criterion, mid-trains CLIP with this objective, and uses the resulting alignment score as a process-level reward in GRPO.

Key results

• CLIP-IRMv2 (ViT-L/14) reaches 88.0 average OOD accuracy across PACS, VLCS, OfficeHome, NICO++, DomainNet, versus 87.4 for Gemini and 86.3 for GPT-4V.
• CLIP-IRMv1 (ViT-B/16) averages 83.6, above the best traditional OOD baseline SWAD at 73.8 and the original CLIP at 83.4.
• Replacing CLIP with CLIP-IRM improves prompt-tuning methods (CoOp, CoCoOp, VPT, MaPLe, PromptSRC) in base-to-new and cross-dataset settings, per the paper's figure-based claims.

Surprising findings

• The OOD baseline IRM (71.6 avg) performs below ERM-style baselines' top results, and proprietary foundation models outperform open-source VLMs on the hardest benchmarks such as DomainNet.
• Open-source VLMs such as BLIP-2 and Qwen-VL show large variance across benchmarks, excelling on PACS/VLCS but dropping sharply on OfficeHome and DomainNet.

Limitations

• The equivalence theorem relies on strong assumptions (diffeomorphic mixing functions, infinite training pairs, optimal encoders) that are not verified empirically.
• Process reward and mid-training details are deferred to appendix/supplement; the mid-training compute and implementation specifics are not given in the main text.

Experimental setup

Datasets: LAION Geometry3K MMK12 PACS VLCS OfficeHome NICO++ DomainNet

Benchmarks: PACS VLCS OfficeHome NICO++ DomainNet Base-to-New generalization Cross-dataset transfer MathVerse MathVision MathVista WeMath HallusionBench

Metrics: accuracy average accuracy reward curves

Baselines: ERM SWAD RSC GroupDRO Fishr CORAL IRM Mixup CLIP BLIP-2 Qwen-VL LLaVA

Builds on: CLIP Invariant Risk Minimization Invariant Policy Optimization GRPO token-level causal representation identifiability

Closest papers by meaning

• Learning Invariant Causal Mechanism from Vision-Language Models (ICML 2025 · similarity 0.91)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.88)
• The Abstraction Gap in Vision-Language Causal Reasoning (ICML 2026 · similarity 0.88)
• On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs (ICML 2026 · similarity 0.88)
• Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective (NeurIPS 2024 · similarity 0.87)
• Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.87)