Research map / Vision-language and multimodal / CLIP and zero-shot
VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes CLIP with hard negatives built from instruction-editing pairs (UltraEdit), a symmetric hard-negative contrastive loss, and RoPE-based long-caption text encoder distilled from CLIP, improving fine-grained perception on MMVP and compositional benchmarks.
Problem
CLIP struggles with fine-grained visual details (color, count, spatial relations) and cannot effectively use long, descriptive captions due to its 77-token absolute positional limit.
Approach
Repurpose instruction-editing datasets (source/target image-caption quadruples) as real-image hard negatives trained with a symmetric image-to-text and text-to-image hard-negative contrastive loss, combined with long InternVL2-generated captions and a RoPE text encoder obtained via distillation from the original CLIP text encoder.
Key results
- With ViT-L/14 at 224, average MMVP accuracy rises from 18.5% to 30.4%, and ImageNet-1K is 76.3% vs 75.5% for OpenAI CLIP.
- With SigLIP2 ViT-SO/14 at 224, average short-caption retrieval improves to 78.8% vs 76.4% and long-caption retrieval to 67.3% vs 62.0%.
- In LLaVA-1.5 with the enhanced ViT-L/14@336 encoder, MME rises from 1510.7 to 1709.0 and MMBench-en from 64.3 to 72.9, while MMVP is 28.0 vs 24.7 (DIVA 31.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with instruction editing data and long captions (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · MMVP | accuracy: 20.7 -> 26.1 | Helped |
| Combine instruction editing data with long-caption data rather than use either alone (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · Short-caption retrieval | average retrieval accuracy: 62.5 -> 73.1 | Helped |
| Use instruction editing data without long-caption data (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · LLaVA-1.5 MMVP | accuracy: 24.7 -> 22.1 | Hurt |
| Use long-caption data without instruction editing data (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · LLaVA-1.5 MMVP | accuracy: 24.7 -> 20.9 | Hurt |
| Add long-caption data to instruction-editing training (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · ImageNet-1K | Top-1 accuracy: 76.6 -> 77.0 | Helped |
| Use RoPE-based text encoder distillation instead of positional interpolation (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · ImageNet-1K | Top-1 accuracy: 70.1 -> 76.3 | Helped |
| Use RoPE-based text encoder distillation instead of positional interpolation (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · Long-caption retrieval | average retrieval accuracy: 78.0 vs. 76.8 | Mixed |
| Use the proposed symmetric hard-negative loss instead of NegCLIP loss (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · ImageNet-1K | Top-1 accuracy: +3.1% | Helped |
| Use the proposed symmetric hard-negative loss instead of triplet loss (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · Short-caption retrieval | average retrieval accuracy: +2.0% | Helped |
| Increase long-caption training data from 18M to 30M pairs (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · Long-caption retrieval | average retrieval accuracy: 76.4 -> 85.1 | Helped |
| Increase long-caption training data from 18M to 30M pairs (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · ImageNet-1K | Top-1 accuracy: 77.0 -> 76.2 | Hurt |
| Use UltraEdit rather than TripletData at 1.4M samples (ablation) | CLIP-IN with OpenAI CLIP ViT-L/14 · Short-caption retrieval | average retrieval accuracy: 70.58 -> 72.9 | Helped |
Surprising findings
- Instruction-editing hard negatives alone or long captions alone reduced MMVP performance in LLaVA-1.5 (22.1 and 20.9 vs 24.7), while their combination improved it to 28.0.
- FG-CLIP (1B long-caption pairs) outperforms the proposed method on long-caption retrieval until the authors scale long-caption data to 30M pairs.
Limitations
- Long-caption retrieval trails FG-CLIP at 18M training pairs versus FG-CLIP's 1B pairs.
- Mixed MMVP gains: ViT-L/14@336 MMVP (26.1) is below DIVA (25.2) only marginally, and some SigLIP2 variants show small or no MMVP gain; LLaVA-1.5 MMVP gain is smaller than DIVA's.
Experimental setup
Datasets: UltraEdit CC3M CC12M COYO LAION TripletData ImageNet-1K Flickr30K MS-COCO DCI ShareGPT4V
Benchmarks: ImageNet-1K Flickr30K MS-COCO DCI ShareGPT4V MMVP ARO Winoground SugarCrepe SPEC POPE MME
Metrics: top-1 accuracy recall@k accuracy group score
Baselines: OpenAI CLIP EVA-CLIP Long-CLIP FineCLIP FG-CLIP DFN-H SigLIP2 DIVA TripletCLIP NegCLIP
Builds on: CLIP NegCLIP TripletCLIP Long-CLIP TULIP CLIP-KD
Closest papers by meaning
- Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling (NeurIPS 2026 · similarity 0.91)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.91)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.90)
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing (ICLR 2026 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.