Research map / Vision-language and multimodal / CLIP and zero-shot
MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLIP, a CLIP variant that adds a DFT-based frequency stage, instance- and token-level frequency/spatial alignment, and guided token merging to improve data efficiency and training speed.
Problem
CLIP uses only one contrastive supervision per image-text pair and retains many non-informative ViT tokens, which makes data use and training inefficient.
Approach
Splits the image encoder into a Fourier-based frequency stage and a spatial attention stage, aligns both frequency and spatial embeddings and tokens with text (token-level via late interaction and KM bipartite matching), and merges tokens with a lightweight Guide ViT's class token to achieve controllable compression.
Key results
- MLIP-ViT-B/32 reaches 41.1% zero-shot ImageNet top-1 versus 43.2% for DeCLIP; with label smoothing it reaches 43.6%.
- With ViT-B/16 pretrained on CC3M for 25 epochs, MLIP gets 18.4% zero-shot ImageNet vs 16.3% for CLIP, at 19.54 GFLOPs vs 19.78.
- On YFCC15M, MLIP-ViT-B/16 reaches 42.9% zero-shot ImageNet vs 37.6% for CLIP (+5.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add frequency transforming to CLIP (ablation) | CLIP ViT-B/16 · MS-COCO image-to-text retrieval | R@1: 10.4 -> 12.7 | Helped |
| Add frequency transforming to CLIP (ablation) | CLIP ViT-B/16 · ImageNet zero-shot | Top-1 accuracy: 16.3 -> 17.6 | Helped |
| Add one-to-many token alignment to CLIP (ablation) | CLIP ViT-B/16 · MS-COCO image-to-text retrieval | R@1: 10.4 -> 13.1 | Helped |
| Add frequency transforming and one-to-one plus one-to-many token alignment (ablation) | CLIP ViT-B/16 · MS-COCO image-to-text retrieval | R@1: 10.4 -> 14.8 | Helped |
| Use one-to-many frequency-text matching and one-to-one spatial-text matching (ablation) | MLIP ViT-B/16 · MS-COCO image-to-text retrieval | R@1: 13.0 -> 14.6 | Helped |
| Add Guide (ablation) | MLIP ViT-B/16 · MS-COCO image-to-text retrieval | R@1: 14.1 -> 14.6 | Helped |
| Add Guide (ablation) | MLIP ViT-B/16 · ImageNet zero-shot | Top-1 accuracy: 18.0 -> 18.4 | Helped |
| Add label smoothing (ablation) | MLIP ViT-B/32 · ImageNet zero-shot | Top-1 accuracy: 41.1 -> 43.6 | Helped |
| Add nearest-neighbor supervision (ablation) | MLIP ViT-B/16 · ImageNet zero-shot | Top-1 accuracy: 46.3 -> 48.2 | Helped |
| Add label smoothing (ablation) | MLIP ViT-B/16 · ImageNet linear probe | Top-1 accuracy: 77.1 -> 78.9 | Helped |
| Pretrain on 95% of CC3M instead of all of CC3M (ablation) | MLIP ViT-B/16 · ImageNet zero-shot | Top-1 accuracy: 18.4 -> 17.9 | Hurt |
| Pretrain on 90% of CC12M instead of all of CC12M (ablation) | MLIP ViT-B/16 · ImageNet zero-shot | Top-1 accuracy: 33.2 -> 31.1 | Hurt |
Surprising findings
- Without label softening, MLIP-ViT-B/32 trails DeCLIP on ImageNet zero-shot (41.1 vs 43.2), and the paper attributes DeCLIP's advantage largely to nearest-neighbor supervision.
- Wall-clock training time for MLIP is about 17 minutes slower than the fastest CLIP, despite fewer GPU hours.
Limitations
- Only ViT-B scale models are studied; ViT-L experiments were not fully run due to compute.
- Performance does not surpass SOTA; the paper prioritizes efficiency and comparisons are hard because baselines are mostly not open-source.
- Token merging makes the encoder harder to transfer to dense tasks such as segmentation.
Experimental setup
Datasets: YFCC15M CC3M CC12M ImageNet CIFAR-10 CIFAR-100 Food-101 Oxford-IIIT Pets Oxford Flowers SUN397 Stanford Cars DTD
Benchmarks: ImageNet CIFAR-10 CIFAR-100 Food-101 Oxford-IIIT Pets Oxford Flowers SUN397 Stanford Cars DTD Caltech-101 FGVC Aircraft Flickr30K
Metrics: zero-shot top-1 accuracy linear-probe top-1 accuracy recall@1/5/10 GFLOPs GPU hours
Baselines: CLIP SLIP FILIP DeCLIP UniCLIP FLIP A-CLIP
Builds on: CLIP ToMe FILIP DeCLIP GFNet
Closest papers by meaning
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.90)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.89)
- AmorLIP: Efficient Language-Image Pretraining via Amortization (NeurIPS 2025 · similarity 0.89)
- Contrastive Localized Language-Image Pre-Training (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.