Research map / Continual learning / Continual fine-tuning of LLMs
CLIP-FMoE: Scalable CLIP via Fused Mixture-of-Experts with Enforced Specialization
Luong Tran, Lan-Cuong Nguyen, Huynh Dang Nguyen, Dat Nguyen Cong, Dung D. Le, Van Nguyen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune CLIP with ICCL experts and Fusion Gate vs vanilla fine-tuning | CLIP ViT-L/14 · zero-shot classification average | top-1 accuracy: 66.08 -> 68.65 | Improved |
| replace CLIP-MoE with CLIP-FMoE | CLIP ViT-L/14 · zero-shot classification average | top-1 accuracy: 66.47 -> 68.65 | Improved |
| replace CLIP-MoE with CLIP-FMoE | CLIP ViT-L/14 · standard retrieval average | recall: 82.29 -> 82.68 | Improved |
| replace TULIP with CLIP-FMoE | CLIP ViT-L/14 · long-context retrieval average | recall: 75.9 -> 77.2 | Improved |
| vanilla fine-tuning of CLIP on captions | CLIP ViT-L/14 · zero-shot classification average | top-1 accuracy: 69.62 -> 66.08 | Hurt |
| unify experts in MoE instead of using single expert 1 (ablation) | CLIP ViT-B/16 · Eurosat | accuracy: 43.33 -> 55.47 | Improved |
| unify experts in MoE instead of using single expert 1 (ablation) | CLIP ViT-B/16 · ImageNet-A | accuracy: 40.64 -> 40.11 | Hurt |
| add Fusion Gate in both stages vs none (ablation) | CLIP-FMoE · zero-shot classification | average top-1 accuracy | Improved |
| add Fusion Gate in stage 1 as well as stage 2 (ablation) | CLIP-FMoE · diverse retrieval | average recall | Improved |
| use CLIP-FMoE parallel training instead of CLIP-MoE sequential | CLIP ViT-L/14 · training time | minutes: 193.51 -> 53.24 | Improved |
| use CLIP-FMoE instead of CLIP-MoE | CLIP ViT-L/14 · inference cost | GFLOPs: 112.44 -> 128.22 | Hurt |
Closest papers by meaning
- MoDE: CLIP Data Experts via Clustering (CVPR 2024 · similarity 0.87)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.87)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.87)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.87)
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.87)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
