Research map / Vision-language and multimodal / CLIP and zero-shot
Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-PGS, a patch masking strategy for CLIP pre-training that pre-selects candidate patches, uses Sobel edge maps and Sinkhorn optimal transport normalization to choose which patches to mask, improving zero-shot, robustness, and compositionality over FLIP, A-CLIP, and E-CLIP at similar cost.
Problem
CLIP pre-training is computationally expensive, and existing patch masking methods (random, attention-based, cluster-based) can remove semantically important image content, hurting visual-text alignment.
Approach
A gradual generation-to-selection masking process: random candidate patches are pre-selected, then Sobel edge maps protect object boundaries and Sinkhorn-based optimal transport normalization of a patch similarity matrix ranks which patches to keep, with masking ratio bounded by a lower and upper limit.
Key results
- CLIP-PGS0.3 reaches 39.5% average zero-shot classification over 17 datasets vs 35.1% for CLIP and 36.9% for E-CLIP, at 0.6x relative training time.
- CLIP-PGS0.3 achieves 38.6% ImageNet-1K zero-shot top-1 vs 36.1% for CLIP, and 64.4% ImageNet-1K linear probing vs 62.3% for CLIP.
- CLIP-PGS0.5 trains at 0.5x relative time and improves average zero-shot accuracy by 4.6 points over FLIP (37.6% vs 33.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLIP-PGS0.3 instead of E-CLIP | ViT-B/16 vision-language pre-training · 17 classification datasets (Average) | top-1 accuracy (%): 36.9 -> 39.5 | Helped |
| use CLIP-PGS0.3 instead of CLIP | ViT-B/16 vision-language pre-training · MS-COCO text retrieval | Recall@1 (%): 34.6 -> 36.0 | Helped |
| use CLIP-PGS0.3 instead of A-CLIP | ViT-B/16 vision-language pre-training · MS-COCO image retrieval | Recall@1 (%): 23.9 -> 25.1 | Helped |
| use CLIP-PGS0.3 instead of E-CLIP | ViT-B/16 vision-language pre-training · CIFAR100 linear probing | top-1 accuracy (%): 69.7 -> 72.3 | Helped |
| use CLIP-PGS0.3 instead of E-CLIP | ViT-B/16 vision-language pre-training · ImageNet-1K robustness (OOD Average) | top-1 accuracy (%): 30.3 -> 31.8 | Helped |
| add edge detection and optimal transport normalization (ablation) | CLIP-PGS0.5 · ImageNet-1K zero-shot classification | top-1 accuracy (%): 35.2 -> 38.0 | Helped |
| add edge detection and optimal transport normalization (ablation) | CLIP-PGS0.3 · ImageNet-1K zero-shot classification | top-1 accuracy (%): 35.9 -> 38.6 | Helped |
| add edge detection (ablation) | CLIP-PGS0.3 without OTN · ImageNet-1K zero-shot classification | top-1 accuracy (%): 35.9 -> 36.8 | Helped |
| add optimal transport normalization (ablation) | CLIP-PGS0.3 without edge detection · ImageNet-1K zero-shot classification | top-1 accuracy (%): 35.9 -> 36.7 | Helped |
| vary the lower masking limit from 0.5 to 0.3 (ablation) | CLIP-PGS · ImageNet-1K zero-shot classification | top-1 accuracy (%): 35.2 -> 35.9; 38.0 -> 38.6 | Helped |
| use ViT-B/16 instead of ViT-S/16 or ViT-B/32 (ablation) | CLIP-PGS0.3 · ImageNet-1K zero-shot classification | top-1 accuracy (%): 35.4; 32.7; 38.6 | Helped |
| increase the initial masking ratio (ablation) | CLIP-PGS · ImageNet-1K zero-shot classification | top-1 accuracy (%) | Hurt |
Limitations
- Pre-trained only on CC12M due to limited compute, so scale effects are untested.
- Experiments only on ViT-B/16; not extended to convolutional networks.
- Baseline results are taken from original papers rather than rerun under identical settings.
Experimental setup
Datasets: CC12M
Benchmarks: Food101 CIFAR10 CIFAR100 SUN397 Cars VOC2007 Aircraft DTD OxfordPets Caltech101 Flowers STL10
Metrics: top-1 accuracy recall@1 recall@5 recall@10 relative training time
Baselines: CLIP FLIP A-CLIP E-CLIP
Builds on: CLIP FLIP Sinkhorn algorithm
Closest papers by meaning
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.