Research map / Architectures and efficiency / Mixture of experts
More Experts Than Galaxies: Conditionally-Overlapping Experts with Biologically-Inspired Fixed Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COMET, a sparse modular architecture that replaces trainable MoE gating with a fixed random projection plus k-winner-take-all cap, so expert overlap depends on input similarity, and evaluates it across MLPs, ViTs, MLP-Mixers, GPT-2 and SARCOS regression.
Problem
Sparse and modular neural networks rely on trainable gating that can cause representation collapse, use disjoint experts that redundantly compute, and often require input or task IDs, limiting flexibility and scale.
Approach
A non-trainable routing network (fixed random matrices followed by k-winner-take-all capping) produces input-dependent binary masks over the backbone's neurons, so every input gets an implicit expert, similar inputs share more active parameters, and the number of possible experts is exponential in layer width.
Key results
- On CIFAR100 with ViT Large, COMET improves accuracy by up to 9% over the standard model, with the gap widening as model size grows.
- On CIFAR10 with a 4-layer MLP, COMET is the top performer once network width is high, while the standard model wins at low capacity.
- Random-input experiments show that similar inputs get more overlapping masks and more similar backbone activations under sparser COMET settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use COMET instead of standard dense routing | ViT-Large · CIFAR100 | accuracy: +9% | Helped |
| use COMET instead of a standard model | large-capacity image-classification models · image classification datasets | accuracy | Helped |
| use COMET instead of a standard GPT model | larger GPT models · WikiText and CodeParrot | validation perplexity | Helped |
| use COMET instead of a standard MLP | MLP · SARCOS | regression performance | Helped |
| add COMET sparsity to a low-capacity model | standard MLP · CIFAR10 | accuracy | Hurt |
| add COMET at 50% sparsity to a smaller GPT model | GPT · language modeling | validation perplexity | Hurt |
| use COMET instead of standard training with a larger learning rate | larger GPT models · WikiText and CodeParrot | validation perplexity | Helped |
| use COMET with reduced batch size | GPT-2 · WikiText | validation perplexity | No clear effect |
| use COMET with reduced batch size | GPT-2 · CodeParrot | validation perplexity | Helped |
| use COMET instead of standard model (ablation) | MLP with GELU, Mish, or SiLU backbone activations · CIFAR10 | performance | Hurt |
| use COMET instead of standard model (ablation) | MLP with most tested backbone activations · CIFAR10 | performance | Helped |
Surprising findings
- Adding more neurons to a standard model (Smaller Model at high capacity) does not improve accuracy, whereas adding neurons under COMET's structured sparsity does.
- COMET underperforms the standard model for non-monotonic activations GELU, Mish, and SiLU at the tested settings.
Limitations
- Unoptimized implementation with no sparse matrix multiplication, so overhead is non-trivial and wall-clock gains are not realized.
- Performance gains are limited or absent for low-capacity models and certain non-monotonic activations (GELU, Mish, SiLU).
- Focuses on single-task learning and out-of-sample generalization; multi-task, transfer and continual learning are left to future work (only a preliminary transfer test is reported).
Experimental setup
Datasets: CIFAR10 SVHN CIFAR100 Tiny ImageNet WikiText CodeParrot SARCOS
Benchmarks: CIFAR10 SVHN CIFAR100 Tiny ImageNet WikiText CodeParrot SARCOS
Metrics: accuracy perplexity validation loss training time GPU memory
Baselines: Standard MLP Smaller model Dropout Trainable routing MLP (cap) Sparse MoE Fixed-routing MoE Example-tied dropout L1 regularization Standard ViT Standard MLP-Mixer GPT-2 standard
Builds on: sparse mixture of experts example-tied dropout neural tangent kernel dropout
Closest papers by meaning
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.87)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.86)
- Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings (NeurIPS 2025 · similarity 0.85)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.85)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.85)
- Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.