Research map / Architectures and efficiency / Mixture of experts
Hard Mixtures of Experts for Large Scale Weakly Supervised Vision
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a single ResNet-18 with 50 hard-routed experts | ResNet-18 · YFCC100M | q@10: 12.41% -> 21.74% | Helped |
| Increase the expert count from 25 to 100 (ablation) | ResNet-18 MoE · YFCC100M | q@10: 19.76% -> 23.82% | Helped |
| Use a shared decoder instead of independent decoders (ablation) | ResNet-18 MoE-50 · YFCC100M | q@10: 21.74% -> 19.70% | Hurt |
| Use a ResNet-34 rather than ResNet-18 expert architecture (ablation) | MoE-50 · YFCC100M | q@10: 21.74% -> 23.76% | Helped |
| Use a four-times-wider ResNet-50 instead of a hard mixture | YFCC100M tag prediction · YFCC100M | q@10: 14.74% vs 21.74% | Hurt |
| Use 50 independent models as an ensemble instead of hard routing | ResNet-18 · YFCC100M | q@10: 13.38% vs 21.74% | Hurt |
| Use a 50-expert mixture | ResNet-18 · ImageNet validation | Top-5 error: 10.69% -> 11.7% | Hurt |
| Use a 50-expert mixture | ResNet-18 · ImageNet validation | Top-1 error: 30.64 -> 30.43 | Helped |
| Pretrain with 50 experts rather than a single ResNet-18 | ResNet-18 features · Flickr transfer | Test accuracy: Actions 53.15 -> 49.69; Flowers 64.76 -> 76.93; Birds 21.28 -> 33.71; MIT SUN 45.01 -> 45.15; Indoors 55.38 -> 57.89; Sp | Mixed |
| Use 50 hard-routed experts | ResNet-18 · Instagram food | q@10: 15.24% -> 21.36% | Helped |
| Use 50 hard-routed experts | ResNet-18 · YFCC100M | Test loss: 7.78 -> 6.93 | Helped |
| Choose experts using true test labels (ablation) | ResNet-34 MoE-50 · YFCC100M | q@10: 23.76% -> 32.8% | Helped |
Closest papers by meaning
- Multi-Task Dense Prediction via Mixture of Low-Rank Experts (CVPR 2024 · similarity 0.88)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.88)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.88)
- Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks (CVPR 2025 · similarity 0.87)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.87)
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.