Research map / Architectures and efficiency / Mixture of experts
Go Wider Instead of Deeper
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WideNet-H instead of ViT-B | ImageNet-1K pretraining · ImageNet-1K | Top-1 accuracy: 78.6 -> 80.1 | Helped |
| Use WideNet-L instead of ViT-MoE-B | ImageNet-1K pretraining · ImageNet-1K | Top-1 accuracy: 77.9 -> 79.5 | Helped |
| Use WideNet-L instead of ViT-MoE-B | CIFAR-10 fine-tuning · CIFAR-10 | accuracy: 98.5 -> 98.8 | Helped |
| Use WideNet 16 experts instead of BERT with factorized embeddings | GLUE and SQuAD fine-tuning · GLUE and SQuAD | Avg: 85.0 -> 85.8 | Helped |
| Increase WideNet experts from 4 to 16 | WideNet · GLUE and SQuAD fine-tuning | Avg: 84.7 -> 85.8 | Helped |
| Use individual Layer Norms instead of shared Layer Norm (ablation) | WideNet-B · ImageNet-1K | Top-1 accuracy: 76.3 -> 77.5 | Helped |
| Use individual Layer Norms instead of shared Layer Norm (ablation) | WideNet-L · ImageNet-1K | Top-1 accuracy: 78.3 -> 79.5 | Helped |
| Use individual Layer Norms instead of shared Layer Norm (ablation) | WideNet-H · ImageNet-1K | Top-1 accuracy: 76.6 -> 80.1 | Helped |
| Use MoE instead of FFN (ablation) | WideNet-L · ImageNet-1K | Top-1 accuracy: 76.9 -> 79.5 | Helped |
| Use MoE instead of FFN (ablation) | WideNet-H · ImageNet-1K | Top-1 accuracy: 79.0 -> 80.1 | Helped |
| Use MoE instead of FFN (ablation) | WideNet-B · ImageNet-1K | Top-1 accuracy | Helped |
| Share parameters across transformer blocks (ablation) | WideNet-B · ImageNet-1K | Top-1 accuracy: 77.5 -> 77.9 | Hurt |
Closest papers by meaning
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.87)
- The Need for Speed: Pruning Transformers with One Recipe (ICLR 2024 · similarity 0.87)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.87)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.87)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.87)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.