Research map / Vision-language and multimodal / CLIP and zero-shot
SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoftCLIP, which replaces CLIP's one-hot image-text targets with softened targets from intra-modal ROI/tag self-similarity, disentangles negatives in the distributions, and uses symmetric KL to improve zero-shot and retrieval performance under noisy web pre-training.
Problem
Strict one-to-one image-text contrastive targets in CLIP treat unpaired samples as fully mutually exclusive, which is harmful when web-crawled pairs (CC3M, CC12M, YFCC15M) contain noise and many-to-many local similarities.
Approach
Use fine-grained intra-modal self-similarity computed from pre-extracted object ROI features and their tags as softened cross-modal targets, mix them with one-hot labels, remove positive logits and renormalize to emphasize negatives in a separate loss, and symmetrize KL divergence in place of cross-entropy.
Key results
- On ImageNet zero-shot with ResNet50, SoftCLIP reaches 24.2 (CC3M) and 43.2 (CC12M) top-1, +6.5 and +7.2 over the CLIP baseline.
- Ablation on CC3M shows label smoothing gives +1.8 and the full SoftCLIP (soft loss, relation-enhanced soft loss, symmetric KL) gives +5.6 over CLIP with ResNet50 (22.1 vs 16.5).
- On YFCC15M-V2 with ResNet50, SoftCLIP improves over CLIP on Oxford/Paris instance retrieval (e.g., Paris Hard 41.0 to 47.0) and copy detection (74.9 to 77.0 mAP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace hard one-hot contrastive alignment with SoftCLIP alignment | CLIP · ImageNet ZS; CC3M/CC12M/YFCC15M; ResNet50, ViT-B/32, ViT-B/16 | top-1 accuracy: 17.7 -> 24.2; 36.0 -> 43.2; 39.6 -> 43.7; 11.9 -> 13.3; 31.5 -> 34.4; 33.1 -> 35.0; 16.9 -> 18.9; 36.8 -> 42.1; 38.9 -> | Helped |
| Replace hard one-hot contrastive alignment with SoftCLIP alignment | CLIP · YFCC15M zero-shot classification AVG; ResNet50 and ViT-B/16 | accuracy: 45.8 -> 49.2; 46.0 -> 49.2 | Helped |
| Replace CLIP with SoftCLIP visual encoder | CLIP · INRIA Copydays strong; YFCC15M; ResNet50 | mAP: 74.9 -> 77.0 | Helped |
| Add ROI/tag soft loss (ablation) | CLIP · ImageNet ZS; CC3M; ResNet50 and ViT-B/32 | top-1 accuracy: +4.0; +1.0 | Helped |
| Add relation-enhanced soft loss (ablation) | CLIP + Soft Loss · ImageNet ZS; CC3M; ResNet50 and ViT-B/32 | top-1 accuracy: 20.5 -> 21.4; 11.7 -> 12.2 | Helped |
| Add symmetric KL (ablation) | CLIP + relation-enhanced soft loss · ImageNet ZS; CC3M; ResNet50 and ViT-B/32 | top-1 accuracy: 21.4 -> 22.1; 12.2 -> 12.5 | Helped |
| Swap R2R and A2A supervision (ablation) | SoftCLIP · ImageNet ZS; CC3M; ResNet50 | top-1 accuracy: 22.1 -> 22.1 | No clear effect |
| Replace R2R/A2A with R2A/A2R (ablation) | SoftCLIP · ImageNet ZS; CC3M; ResNet50 | top-1 accuracy: 22.1 -> 21.9 | Hurt |
| Replace R2R/A2A with A2R/R2A (ablation) | SoftCLIP · ImageNet ZS; CC3M; ResNet50 | top-1 accuracy: 22.1 -> 21.8 | Hurt |
| Use mean ROI aggregation instead of MHSA (ablation) | SoftCLIP · ImageNet ZS; CC3M; ResNet50 and ViT-B/32 | top-1 accuracy: 22.1 -> 21.8; 12.5 -> 11.9 | Hurt |
| Use max ROI aggregation instead of MHSA (ablation) | SoftCLIP · ImageNet ZS; CC3M; ResNet50 and ViT-B/32 | top-1 accuracy: 22.1 -> 21.8; 12.5 -> 12.2 | Hurt |
| Use min ROI aggregation instead of MHSA (ablation) | SoftCLIP · ImageNet ZS; CC3M; ResNet50 and ViT-B/32 | top-1 accuracy: 22.1 -> 19.2; 12.5 -> 10.4 | Hurt |
Surprising findings
- Using image-to-image and text-to-text self-similarity as soft targets did not perform as well as ROI and tag self-similarity, which the authors attribute to inaccurate early-stage similarities and the coarser global view.
- Setting the soft-label mixing weight beta to 0.0001 still performs only slightly worse, which the authors attribute to the relation-enhanced negative-only loss recovering the negatives.
Limitations
- Depends on a pre-trained object-attribute detector (VinVL) to extract ROI features and tags, adding preprocessing cost and dependency.
- Gains are reported mainly with small pre-training datasets (CC3M, CC12M, YFCC15M) and models trained from scratch; scaling to larger data is not shown.
Experimental setup
Datasets: CC3M CC12M YFCC15M-V2 ImageNet Pets DTD Food-101 Flowers-102 SUN397 Caltech-101 Flickr30K MS-COCO
Benchmarks: ImageNet Pets DTD Food-101 Flowers-102 SUN397 Caltech-101 Flickr30K MS-COCO Oxford revisited Paris revisited INRIA Copydays
Metrics: zero-shot top-1 accuracy recall@K mAP
Baselines: CLIP PyramidCLIP (label smoothing) DeCLIP
Builds on: CLIP PyramidCLIP label smoothing VinVL DeCLIP
Closest papers by meaning
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.89)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.88)
- Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.