Research map / Vision-language and multimodal / Cross-modal learning
SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies an "affinity inconsistency" between image and text sample similarities in contrastive VLP and adds a SaCo loss that minimizes the L1 gap between image and text similarity vectors within a batch, usable from scratch (with pseudo-affinity mimicking) or as continued pre-training.
Problem
Contrastive vision-language pre-training aligns paired image-text embeddings but ignores whether sample-to-sample affinities agree across modalities, so image-image and text-text similarities diverge, hurting retrieval and classification.
Approach
An additional loss that directly minimizes the L1 distance between each sample's image-similarity vector and text-similarity vector within a batch, complementing the contrastive objective; stabilized from scratch via pseudo-affinity mimicking from a well-trained vision model, or applied by continued pre-training on a converged CLIP.
Key results
- From-scratch CC3M pre-training with SaCo raises zero-shot ImageNet-1K top-1 for CLIP ViT-B/32 from 11.9 to 18.3 (+6.4) and Flickr30K I2T R@1 from 16.0 to 25.3 (+9.3).
- Continued pre-training of OpenAI CLIP ViT-B/32 on LLaVA-595K improves zero-shot ImageNet-1K top-1 from 63.0 to 64.0 and Flickr30K I2T R@1 from 77.8 to 81.7.
- Pseudo-affinity mimicking from a well-trained model is needed for stable from-scratch training; applying mimicking alone yields only minor gains and hurts in continued pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SaCo loss and pseudo-affinity mimicking during scratch pre-training | CLIP ViT-B/32 · ImageNet-1K | Top-1 accuracy: 11.9 -> 18.3 (+6.4) | Helped |
| Add SaCo loss and pseudo-affinity mimicking during scratch pre-training | CLIP R50 · Flickr30K, I2T | Recall@1: 29.2 -> 42.0 (+12.8) | Helped |
| Add SaCo loss and pseudo-affinity mimicking during scratch pre-training | CLIP R50 · MS-COCO, T2I | Recall@1: 13.3 -> 16.0 (+2.7) | Helped |
| Continue pre-training with SaCo loss | OpenAI CLIP ViT-B/16 · Flickr30K, I2T | Recall@1: 81.7 -> 85.5 (+3.8) | Helped |
| Use SaCo loss without pseudo-affinity mimicking instead of the full scratch-training method (ablation) | CLIP ViT-B/32 · ImageNet-1K | Top-1 accuracy: 18.3 -> 17.5 | Hurt |
| Use pseudo-affinity mimicking without SaCo loss instead of the full scratch-training method (ablation) | CLIP ViT-B/32 · Flickr30K, I2T | Recall@1: 25.3 -> 17.8 | Hurt |
| Add pseudo-affinity mimicking without SaCo loss (ablation) | CLIP ViT-B/32 during continuous pre-training · ImageNet-1K | Top-1 accuracy: 63.0 -> 60.8 (-2.2) | Hurt |
| Add pseudo-affinity mimicking to SaCo during continuous pre-training (ablation) | CLIP ViT-B/32 · Flickr30K, I2T | Recall@1: 77.8 -> 78.6 (+0.8) | Mixed |
| Mimic visual pseudo-affinity rather than language pseudo-affinity (ablation) | CLIP ViT-B/16 scratch pre-training · ImageNet-1K | Top-1 accuracy: 20.7 -> 21.8 | Helped |
| Use DINOv2 ViT-L/14 rather than CLIP ViT-L/14 for visual pseudo-affinity (ablation) | CLIP ViT-B/16 scratch pre-training · ImageNet-1K | Top-1 accuracy: 21.8 -> 21.6 | No clear effect |
Surprising findings
- Pseudo-affinity mimicking alone gives little benefit, and in continued pre-training it degrades performance (-2.2 ImageNet top-1), so the SaCo term is the key contributor.
- Gains on image-to-text retrieval are roughly 1-2 times larger than on text-to-image retrieval, suggesting the text embedding space benefits more.
Limitations
- From-scratch training is unstable early on because affinities are noisy, requiring pseudo-affinity mimicking from an external model.
- Pseudo-affinity mimicking depends on a separate well-trained model (CLIP ViT-L/14 or DINOv2) as teacher.
Experimental setup
Datasets: CC3M YFCC15M LLaVA-595K
Benchmarks: ImageNet-1K Flickr30K MS-COCO
Metrics: Top-1 accuracy Top-5 accuracy Recall@1 Recall@5 Recall@10 Pearson correlation (affinity consistency)
Baselines: CLIP BLIP DeCLIP SLIP SoftCLIP
Builds on: CLIP DINOv2
Closest papers by meaning
- FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · similarity 0.90)
- Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples (AAAI 2025 · similarity 0.89)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.89)
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- MAFA: Managing False Negatives for Vision-Language Pre-training (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.