Research map / Vision-language and multimodal / CLIP and zero-shot
Extracting Multimodal Learngene in CLIP: Unveiling the Multimodal Generalizable Knowledge
Ruiming Chen, Junming Yang, Shiyu Xia, Xu Yang, Xin Geng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the extracted multimodal block to the initialization (ablation) | MM-LG · Flickr30k retrieval I2T (12-layer) | I2T recall: 23.47 -> 65.18 | Improved |
| add the extracted unimodal blocks to the initialization (ablation) | MM-LG · Flickr30k retrieval I2T (12-layer) | I2T recall: 23.37 -> 65.18 | Improved |
| add the extracted multimodal block to the initialization (ablation) | MM-LG · CIFAR-100 (12-layer) | accuracy: 78.11 -> 87.70 | Improved |
| add the extracted unimodal blocks to the initialization (ablation) | MM-LG · CIFAR-100 (12-layer) | accuracy: 81.15 -> 87.70 | Improved |
| add the extracted multimodal block to the initialization (ablation) | MM-LG · Flickr30k retrieval I2T (6-layer) | I2T recall: 13.12 -> 51.09 | Improved |
| replace TLEG linear expansion with the multimodal block learngene | MM-LG vs TLEG · Flickr30k retrieval I2T (12-layer) | I2T recall: 61.05 -> 65.18 | Improved |
| replace TLEG with MM-LG | TLEG · COCO image captioning (6-layer) | CIDEr: 39.87 -> 49.80 | Improved |
| replace PT-FT with MM-LG initialization | PT-FT · COCO image captioning (12-layer) | CIDEr: 53.90 -> 59.34 | Improved |
| replace PT-FT with MM-LG initialization | PT-FT · Flickr30k retrieval I2T (6-layer) | I2T recall: 52.76 -> 51.09 | Hurt |
| replace PT-FT with MM-LG initialization | PT-FT · Flickr30k retrieval I2T (12-layer) | I2T recall: 61.53 -> 65.18 | Improved |
| use extracted learngene instead of per-scale pre-training (storage) | PT-FT · Parameter storage for three scales | Params (M): 151.4 -> 37.4 | Improved |
| use one learngene extraction instead of per-scale pre-training (cost) | PT-FT · Pre-training cost | GPU-hours: 1142.4 -> 407.8 | Improved |
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing (ICLR 2026 · similarity 0.89)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.88)
- Reproducible Vision-Language Models Meet Concepts Out of Pre-Training (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
