Research map / Vision-language and multimodal / CLIP and zero-shot
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the dense one-tower encoder with LIMoE sparse experts | L/16 contrastive image-text model · ImageNet zero-shot | accuracy: 73.5 -> 78.6 | Helped |
| Use LIMoE-H/14 with additional JFT data | ImageNet zero-shot classification · ImageNet | accuracy: 76.2 -> 84.1 | Helped |
| Add global entropy regularization for text routing (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 54.3 -> 56.2 | Helped |
| Add local entropy regularization for text routing (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 53.3 -> 56.2 | Helped |
| Add local entropy regularization for image routing (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 56.2 -> 53.5 | Hurt |
| Add global entropy regularization for image routing (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 56.0 -> 56.2 | No clear effect |
| Replace modality-conditioned local and global entropy losses with a modality-expert mutual-information loss (ablation) | LIMoE · ImageNet zero-shot | accuracy: -1.5% | Hurt |
| Select five experts per token using target entropy loss and summed BPR (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 55.5 -> 60.4 | Helped |
| Increase the available expert pool from 4 to 64 (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 52.3 -> 58.0 | Helped |
| Use batch priority routing instead of shuffled dispatch (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 37.6 -> 56.8 | Helped |
| Replace the shared router with per-modality or partitioned routing (ablation) | LIMoE-B/16 · ImageNet zero-shot | accuracy: 56.8 -> 56.8 | No clear effect |
| Use entropy losses rather than classic auxiliary losses (ablation) | LIMoE-B/16 with image sequence length 196 · ImageNet zero-shot | accuracy: 17.5 -> 57.2 | Helped |
Closest papers by meaning
- Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders (ICML 2025 · similarity 0.89)
- CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.87)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.87)
- Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.