Research map / Vision-language and multimodal / Multimodal LLMs
CF-VLM：CounterFactual Vision-Language Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CF-VLM, a fine-tuning framework that adds counterfactual image-text pairs (generated with an LLM and SDXL) and three contrastive losses to CLIP and Qwen-VL to improve compositional reasoning and reduce hallucination.
Problem
Vision-language models rely on superficial correlations and struggle with fine-grained discrimination and causal reasoning, failing on minimal semantic edits to image-text pairs.
Approach
Uses counterfactual image-text pairs (jointly edited scenarios and minimally edited images) with three objectives: a symmetric InfoNCE alignment loss, a hinge loss separating anchor from complete counterfactual scenes, and a hard-negative hinge loss against the most similar minimally edited image.
Key results
- On ConMe with CLIP ViT-B/32 (CC12M), CF-VLM reaches 59.13% average versus 54.43% for standard fine-tuning and 56.53% for TripletCLIP.
- On ConMe with Qwen-VL 7B, CF-VLM reaches 87.57% average, +4.9 over standard fine-tuning (82.6%).
- On ARO with CLIP ViT-B/32, CF-VLM averages 89.35% versus 87.75% for COGT-CLIP (the strongest listed baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with CF-VLM counterfactual supervision rather than TextNeg FT | Qwen-VL 7B · ConMe (Avg) | accuracy: 84.13 -> 87.57 | Helped |
| Fine-tune with CF-VLM counterfactual supervision rather than TextNeg FT | Qwen-VL 7B · ARO (Avg) | accuracy | Helped |
| Fine-tune with CF-VLM counterfactual supervision rather than TextNeg FT | Qwen-VL 7B · VL-Checklist (Avg) | accuracy: 88.43 -> 90.57 | Helped |
| Use joint image-text counterfactuals instead of original pairs and random negatives (ablation) | CF-VLM training · ConMe (Avg) | accuracy: 54.4 -> 59.1 | Helped |
| Use joint image-text counterfactuals rather than text-only counterfactuals (ablation) | CF-VLM training · ConMe (Avg) | accuracy: 55.9 -> 59.1 | Helped |
| Use joint image-text counterfactuals rather than image-only counterfactuals (ablation) | CF-VLM training · ConMe (Avg) | accuracy: 56.1 -> 59.1 | Helped |
| Add fine-grained causal discrimination loss (ablation) | CF-VLM with alignment and scenario discrimination losses · ConMe (Avg) | accuracy: 57.9 -> 59.1 | Helped |
| Add foundational cross-modal alignment loss (ablation) | CF-VLM with scenario and fine-grained causal discrimination losses · ConMe (Avg) | accuracy: 57.3 -> 59.1 | Helped |
| Add counterfactual scenario discrimination loss (ablation) | CF-VLM with alignment and fine-grained causal discrimination losses · ConMe (Avg) | accuracy: 58.0 -> 59.1 | Helped |
| Add attribute-level counterfactuals (ablation) | CF-VLM with ViT-B/32 · ConMe Replace-Attr | accuracy: 58.2 -> 60.4 | Helped |
| Add object-level counterfactuals (ablation) | CF-VLM with ViT-B/32 · ConMe Replace-Obj | accuracy: -2.9 points when removed | Helped |
| Add relation-level counterfactuals (ablation) | CF-VLM with ViT-B/32 · ConMe Replace-Rel | accuracy: -3.3 points when removed | Helped |
Surprising findings
- Reported numbers are inconsistent across sections: batch ratio is stated as 1:1 in one place and 1:4 in another, and the counterfactual generator is Qwen2-72B in one section and LLaMA-3-70B in another.
- Zero-shot ImageNet-1k top-1 accuracy is reported as only 8.83%, far below typical CLIP ViT-B/32 zero-shot results.
Limitations
- Counterfactual data quality depends on LLM and diffusion generators, with no human validation reported.
- Hallucination gains on POPE and MME are described as marginal.
- Several inconsistencies in hyperparameters and generator choice across sections make the setup hard to reproduce.
Experimental setup
Datasets: CC12M CC3M MSCOCO Captions
Benchmarks: ConMe ARO VL-Checklist ImageNet-1k MSCOCO Flickr30k POPE MME
Metrics: accuracy top-1 accuracy top-5 accuracy Recall@1 Recall@5 precision recall F1
Baselines: NegCLIP TripletCLIP CE-CLIP+ Structure-CLIP COGT-CLIP Standard FT TextNeg FT LLaVA-1.5 InstructBLIP MiniGPT-4
Builds on: CLIP TripletCLIP NegCLIP Qwen-VL SDXL
Closest papers by meaning
- Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability (NeurIPS 2026 · similarity 0.90)
- Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency (NeurIPS 2026 · similarity 0.89)
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (CVPR 2024 · similarity 0.89)
- CFPO: Counterfactual Policy Optimization for Multimodal Reasoning (ICML 2026 · similarity 0.89)
- Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework (CVPR 2026 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.