Research map / Vision-language and multimodal / CLIP and zero-shot
Improving fine-grained understanding in image-text pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPARC, a sparse fine-grained contrastive objective that groups image patches per caption token via sparsified, min-max-normalized token-patch similarities, and pretrains ViTs from scratch on ALIGN/JFT/LTIP with a like-for-like comparison against CLIP and fine-grained baselines.
Problem
Contrastive image-text pretraining on global embeddings discards fine-grained visual detail, while existing local/fine-grained losses are computationally expensive, unstable, rely on pretrained encoders, and use softmax weights that bias toward one-to-one token-patch matching.
Approach
For each caption token, build a language-grouped vision embedding as an alignment-weighted sum of patches, where weights come from a sparsified (threshold 1/P) min-max-normalized token-patch similarity rather than softmax; contrast these with token embeddings within each image-text pair only, combined with a global CLIP-style loss.
Key results
- SPARC ViT-B/16 zero-shot ImageNet top-1 72.6% vs CLIP 71.6%; ImageNet-A 33.8 vs 32.2.
- SPARC ViT-B/16 MSCOCO image-to-text R@1 57.6 vs CLIP 56.2; text-to-image R@1 43.0 vs 42.4.
- Zero-shot segmentation on Pascal VOC mIoU 27.36 vs CLIP 23.02; LVIS AP_rare 25.1 vs CLIP 22.0 with ViT-B/16 detection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPARC instead of CLIP | ViT-B/16 image-text pretraining · ImageNet zero-shot classification | Top-1 accuracy: 71.6 -> 72.6 | Helped |
| Use SPARC instead of CLIP | ViT-B/16 image-text pretraining · Flickr30k image-to-text retrieval | R@1: 84.0 -> 84.4 | Helped |
| Use SPARC instead of CLIP | OWL-ViT ViT-B/16 backbone pretraining · LVIS rare-class detection | AP rare: 22.0 \pm 0.79 -> 25.1 \pm 0.95 | Helped |
| Use SPARC instead of CLIP | ViT-B/16 image-text pretraining · Pascal VOC zero-shot segmentation | mIoU: +4.34 mIoU | Helped |
| Use SPARC instead of CLIP | Flamingo-style model vision backbone · MSCOCO captioning | CIDEr: 24.3 -> 25.3 | Helped |
| Add sparsity to the similarity matrix (ablation) | SPARC ViT-B/16 · MSCOCO image-to-text retrieval | R@1: 56.1 -> 57.6 | Helped |
| Add sparsity to the similarity matrix (ablation) | SPARC ViT-B/16 · MSCOCO text-to-image retrieval | R@1: 42.4 -> 43.0 | Helped |
| Add sparsity to the similarity matrix (ablation) | SPARC ViT-B/16 · ImageNet zero-shot classification | Top-1 accuracy: 72.1 -> 72.6 | Helped |
| Use sparse normalized similarities instead of softmax alignment weights (ablation) | SPARC ViT-B/16 · MSCOCO image-to-text retrieval | R@1: 55.2 -> 57.6 | Helped |
| Use sparse normalized similarities instead of softmax alignment weights (ablation) | SPARC ViT-B/16 · MSCOCO text-to-image retrieval | R@1: 41.6 -> 43.0 | Helped |
| Use sparse normalized similarities instead of softmax alignment weights (ablation) | SPARC ViT-B/16 · ImageNet zero-shot classification | Top-1 accuracy: 70.6 -> 72.6 | Helped |
Surprising findings
- FILIP was highly unstable to train from scratch across learning rates and weight decays, underperforming CLIP substantially in the like-for-like setting.
- Removing softmax in favor of sparse min-max weights matters more than sparsity alone; softmax ablation caused the largest drop (ImageNet 70.6 vs 72.6).
Limitations
- Pretraining uses large proprietary-scale datasets (ALIGN, JFT, LTIP) that are not public, limiting reproducibility.
- SPARC does not consistently win: e.g., CLIP is best on some ImageNet-C and retrieval R@5/R@10 metrics; ViT-L ImageNet-A is lower than CLIP.
- Baselines were reimplemented by the authors with modifications (e.g., no GLoRIA word aggregation, batch size 4096 for GLoRIA), which may affect comparisons.
Experimental setup
Datasets: ALIGN JFT LTIP Objects365 Visual Genome VTP
Benchmarks: ImageNet ImageNet-V2 ImageNet-R ImageNet-C ImageNet-A ImageNet-Sketch MSCOCO Flickr30k LVIS Pascal VOC Pascal Context
Metrics: top-1 accuracy recall@k mean average precision mIoU CIDEr K-Precision TFLOPS peak memory
Baselines: CLIP FILIP PACL GLoRIA MGCA
Builds on: CLIP FILIP GLoRIA MGCA PACL
Closest papers by meaning
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.90)
- Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.89)
- SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment (ICML 2024 · similarity 0.89)
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.