Research map / Vision-language and multimodal / CLIP and zero-shot
SmartCLIP: Modular Vision-language Alignment with Identification Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SmartCLIP, which learns adaptive binary masks over CLIP embedding dimensions so each caption aligns only with its relevant concept block, backed by a block-wise identifiability theory for multi-caption image-text data.
Problem
CLIP suffers from information misalignment when an image has multiple partial captions and from entangled representations when trained on long detailed captions, hurting both short and long text retrieval.
Approach
Frame image-text alignment as latent-variable identification with a caption-dependent binary mask over latent dimensions, learned via a sparsity-regularized masking network and a modular contrastive loss, which recovers unions/intersections of concept blocks without explicit view-group labels.
Key results
- On Urban1k, SmartCLIP raises long text-to-image R@1 from 78.9% (Long-CLIP) to 90.0% with ViT-B/16.
- On ShareGPT4V long text-to-image retrieval, SmartCLIP reaches 98.7% R@1 (ViT-B/16) versus 78.2% for CLIP.
- On ViT-L/14 zero-shot classification, SmartCLIP is best on GTSRB (52.4 vs 50.2 CLIP) and VOC2007-Multi (83.7 vs 79.0 CLIP), but lower on ImageNet (72.5 vs 75.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use modular selective alignment instead of standard contrastive alignment (ablation) | SmartCLIP · MSCOCO and GPT4V retrieval | R@1 | Helped |
| Add sparsity to the mask network (ablation) | SmartCLIP · MSCOCO and GPT4V retrieval | R@1 | Helped |
| Vary the alignment coefficient λalign across the tested range (ablation) | SmartCLIP · MSCOCO and GPT4V retrieval | R@1 | No clear effect |
| Use 3 captions per image instead of 1 (ablation) | SmartCLIP trained with COCO captions · Flickr30K retrieval | R@1: 53.6 -> 53.6; 39.3 -> 40.9 | Mixed |
| Use 5 captions per image instead of 1 (ablation) | SmartCLIP trained with COCO captions · Flickr30K retrieval | R@1: 53.6 -> 56.4; 39.3 -> 41.2 | Helped |
| Use 5 captions per image instead of 1 (ablation) | SmartCLIP trained with COCO captions · ShareGPT-val long-text retrieval | R@1: 85.3 -> 85.2; 89.3 -> 86.5 | Mixed |
| Combine ShareGPT training data with COCO (ablation) | SmartCLIP · Flickr30K and ShareGPT-val retrieval | R@1: 55.6 -> 57.0; 36.3 -> 38.4; 98.7 -> 97.8; 98.1 -> 98.5 | Mixed |
| Fine-tune on ShareGPT4V instead of using the pretrained CLIP model | CLIP ViT-L/14 · Zero-shot classification | accuracy | Mixed |
| Use SmartCLIP instead of Long-CLIP as the SDXL text encoder | SDXL · ShareGPT4V validation, long-caption image generation | DINO-L: 0.401 -> 0.414 | Helped |
Surprising findings
- Standard contrastive learning without the modular objective performs much worse, because the mask makes positives easy to separate from negatives.
- Adding more captions per image improves Flickr30K retrieval but degrades long-text retrieval.
Limitations
- Condition 4.2-ii (full support of joint distribution) may be violated when some images have few captions.
- Zero-shot classification drops on ImageNet and some other datasets relative to CLIP.
Experimental setup
Datasets: ShareGPT4V COCO
Benchmarks: ShareGPT4V validation Urban1k COCO2017 5k Flickr30K ImageNet GTSRB Country211 Fer2013 Fgvc-aircraft ImageNet-V2 VOC2007 VOC2007-Multi
Metrics: R@1 R@5 R@10 top-1 accuracy KID Precision Recall F1 DINO-L
Baselines: CLIP Long-CLIP Direct Fine-tuning
Builds on: CLIP Long-CLIP multi-view latent variable identification straight-through estimator
Closest papers by meaning
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.92)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.92)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.91)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.91)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.90)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.