Research map / Vision-language and multimodal / CLIP and zero-shot
AmorLIP: Efficient Language-Image Pretraining via Amortization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AmorLIP, a CLIP pretraining framework that amortizes the contrastive partition function with lightweight MLPs trained on rolling minibatches, using a spectral (random Fourier feature) representation and f-divergence or l2-log objectives.
Problem
CLIP's contrastive objective needs very large batches (tens of thousands of samples) and many GPUs because partition-function estimates depend on in-batch negatives, creating estimation bias and inter-sample dependency.
Approach
Reformulate CLIP as energy-based learning, express the partition function linearly via random Fourier features of a Gaussian kernel, and learn small MLPs (in log-space) to amortize the partition function across rolling minibatches, alternating amortization and encoder updates.
Key results
- AmorLIP (l2-log) reaches 18.92% on ImageNet and shifts and 24.11% average over 38 tasks with ResNet-50 on CC3M, vs 21.48% for CLIP (up to 12.24% relative gain).
- With ViT-B/32 on CC12M, AmorLIP (l2-log) achieves 30.66% average over 38 tasks vs 27.65% for CLIP and 29.00% for FastCLIP.
- Lightweight amortization adds only 0.23% training time and 0.26% VRAM overhead in the most efficient configuration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AmorLIP instead of the strongest baseline, FastCLIP | ResNet-50 pretrained on CC3M · DataComp 38-task average | average performance: 23.46 -> 24.11 | Helped |
| Use AmorLIP instead of the strongest baseline, FastCLIP | ViT-B/32 pretrained on CC12M · DataComp 38-task average | average performance: 29.00 -> 30.66 | Helped |
| Use l2-log rather than f-div amortization (ablation) | AmorLIP · DataComp 38-task average, both training scales | average performance: 24.08 -> 24.11; 29.91 -> 30.66 | Helped |
| Use f-div rather than l2-log amortization (ablation) | AmorLIP · Retrieval, ResNet-50 pretrained on CC3M | recall@1: 27.34 -> 27.61 | Helped |
| Amortize positive and negative samples rather than negative samples only (ablation) | AmorLIP · IN & shifts, medium-scale | top-1 accuracy: 18.02 -> 18.92 | Helped |
| Set amortization update frequency to T_online^-1=1/8 rather than 1 (ablation) | AmorLIP · IN & shifts, medium-scale | top-1 accuracy: 18.28 -> 18.92 | Helped |
| Reduce amortization update frequency from T_online^-1=1/8 to 1/32 (ablation) | AmorLIP · Retrieval, medium-scale | recall@1: 28.10 -> 28.39 | Helped |
| Set amortization network dimension factor f_d to 0.5 rather than 2 (ablation) | AmorLIP · DataComp 38-task average, medium-scale | average performance: 23.97 -> 24.11 | Helped |
| Increase target EMA factor (ablation) | AmorLIP · Classification and overall performance | performance | Helped |
| Set previous-epoch combination weight β_T to 0.8 rather than 1.0 (ablation) | AmorLIP · Overall performance, medium-scale | performance | Helped |
Surprising findings
- Less frequent amortization-network updates and smaller amortization networks generally gave equal or better downstream performance than larger ones, which the authors attribute to reduced overfitting.
Limitations
- Experiments limited to datasets up to ~10M samples and ViT-B/32 scale; no billion-scale validation.
- The l2-log objective is biased (second-order approximation of KL), trading bias for lower variance.
- No error bars or statistical significance reported.
Experimental setup
Datasets: CC3M CC12M DataComp
Benchmarks: ImageNet-1k ImageNet-Sketch ImageNet-V2 ImageNet-A ImageNet-O ImageNet-R ObjectNet Flickr30k MS-COCO DataComp 38 tasks
Metrics: top-1 zero-shot accuracy recall@1 average over 38 tasks per-step training time VRAM usage
Baselines: CLIP SigLIP SogCLR FastCLIP
Builds on: CLIP SogCLR FastCLIP random Fourier features MoCo
Closest papers by meaning
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.89)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- OT-CLIP: Understanding and Generalizing CLIP via Optimal Transport (ICML 2024 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.