Research map / Vision-language and multimodal / CLIP and zero-shot
VladVA: Discriminative Fine-tuning of LVLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts a generative LLaVA-1.5-7B LVLM into a discriminative image-text embedding model by combining contrastive loss on short captions with next-token prediction on long captions, using soft prompts plus LoRA.
Problem
Contrastive VLMs like CLIP show weak language understanding (bag-of-words behavior), while LVLMs have strong vision-language reasoning but are autoregressive and poorly suited to discriminative image-text tasks.
Approach
Adapt a generative LVLM into a two-tower discriminative model using summary tokens from handcrafted prompts, training with a contrastive loss on short captions and an autoregressive loss on long captions, with parameter-efficient soft prompts and LoRA adapters.
Key results
- Zero-shot image retrieval R@1 on Flickr30k of 85.0% vs 83.3% for EVA-CLIP (18B) and 79.5% for E5-V (LLaVA-Next-8B).
- SugarCrepe object swap accuracy of 79.0 vs 65.3 for EVA-CLIP (18B), with relation replacement 86.8 vs 76.1.
- Scaling training data from 1M to 8.1M samples yields steady gains (e.g., SugarCrepe Replace 87.1 to 92.3 with the 1M-to-2.8M CC3M additions).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaVA-1.5-7B with VladVA | LLaVA-1.5-7B · Flickr30K image retrieval | R@1: 83.3 -> 85.0 | Helped |
| fine-tune LLaVA-1.5-7B with VladVA | LLaVA-1.5-7B · COCO image retrieval | R@1: 55.6 -> 59.0 | Helped |
| fine-tune LLaVA-1.5-7B with VladVA | LLaVA-1.5-7B · nocaps image retrieval | R@1: 69.3 -> 72.3 | Helped |
| add soft prompting (ablation) | LLaVA-1.5-7B · Flickr30K | T2I R@1: 59.6 -> 76.7 | Helped |
| add LoRA adapter fine-tuning (ablation) | LLaVA-1.5-7B · Flickr30K | T2I R@1: 59.6 -> 79.1 | Helped |
| add next-token prediction loss on long captions (ablation) | LLaVA-1.5-7B with adapter fine-tuning and soft prompting · Flickr30K | T2I R@1: 79.6 -> 80.6 | Helped |
| add LoRA adapter fine-tuning, soft prompting, and next-token prediction loss (ablation) | LLaVA-1.5-7B · SugarCrepe | Swap: 59.8 -> 75.5 | Helped |
| add LoRA adapter fine-tuning, soft prompting, and next-token prediction loss (ablation) | LLaVA-1.5-7B · SugarCrepe | Replace: 81.9 -> 89.5 | Helped |
| add LoRA adapter fine-tuning, soft prompting, and next-token prediction loss (ablation) | LLaVA-1.5-7B · SugarCrepe | Add: 64.7 -> 89.5 | Helped |
| scale training data from OpenImages (1M) to OpenImages (4M) (ablation) | VladVA · SugarCrepe | Swap: 72.0 -> 79.6 | Helped |
| add ShareGPT-4V (1.3M) training data (ablation) | VladVA · SugarCrepe | Swap: 79.6 -> 80.3 | Helped |
| add CC3M (2.8M) training data (ablation) | VladVA · SugarCrepe | Swap: 80.3 -> 80.9 | Helped |
Surprising findings
- Using long captions with the contrastive loss alone collapses training (loss goes to zero quickly); long captions help only with the autoregressive loss.
- Soft prompts alone perform surprisingly well; decoded soft prompts remain nearly identical to the handcrafted prompts.
Limitations
- Text-to-text compositionality lags behind E5-V, which uses text-text contrastive training.
- Experiments are at relatively small scale (up to 8.1M samples) with mostly one LVLM backbone (LLaVA-1.5-7B) in the main text.
Experimental setup
Datasets: OpenImages CC3M ShareGPT-4V Flickr30k COCO nocaps SugarCrepe SugarCrepe++
Benchmarks: Flickr30k COCO nocaps SugarCrepe SugarCrepe++
Metrics: R@1 R@10 accuracy
Baselines: CLIP (ViT-L) BLIP (ViT-L) BLIP2 (ViT-L) OpenCLIP (ViT-G/14) OpenCLIP (ViT-BigG/14) EVA-02-CLIP (ViT-E/14+) EVA-CLIP (8B) EVA-CLIP (18B) E5-V (LLaVA-Next-8B) E5-V (LLaVA-1.5-7B) VLM2Vec (Mistral-7B) NegCLIP
Builds on: LLaVA-1.5 CLIP E5-V soft prompting LoRA
Closest papers by meaning
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.92)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.91)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.