Research map / Architectures and efficiency / Mixture of experts
Brain-Like Processing Pathways Form in Models With Heterogeneous Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends a Heterogeneous Mixture-of-Experts with a complexity-based routing cost, task-performance scaling, and expert dropout to form brain-like processing pathways that match cortical-subcortical and multiple-demand system patterns on 82 Mod-Cog cognitive tasks.
Problem
Heterogeneous regions do not automatically form stable, task-relevant processing pathways in Mixture-of-Experts models, and it is unclear what inductive biases yield brain-like pathway formation.
Approach
Three biologically motivated inductive biases (a routing cost penalizing use of larger GRU experts, scaling that cost by task performance, and randomized low-weight expert dropout) turn a heterogeneous HMoE into a Mixture-of-Pathways model whose routing is consistent, self-sufficient, and distinct across tasks.
Key results
- Mixture-of-Pathways routing consistency across 20 runs rises to mean pairwise correlation 0.71 with routing cost scaled by task performance, versus 0.0324 for the baseline.
- With expert dropout (beta=0.8), accuracy after blocking experts with w<0.025 drops only from 86.5% to 77.7%, versus 98.2% to 15.1% without dropout.
- Pathway complexity correlates positively with task difficulty in the full model (r=0.54 in Table 1), while the baseline shows no significant relationship (r=-0.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add routing cost to penalize complex experts | Baseline HMoE · Mod-Cog tasks | routing consistency (mean pairwise correlation): 0.0324 -> 0.15 | Helped |
| scale the routing cost by task response loss | HMoE with routing cost · Mod-Cog tasks | routing consistency (mean pairwise correlation): 0.15 -> 0.71 | Helped |
| add expert dropout (ablation) | Mixture-of-Pathways · Mod-Cog tasks | routing consistency (mean pairwise correlation): 0.71 -> 0.51 | Hurt |
| add expert dropout (ablation) | Mixture-of-Pathways · Mod-Cog tasks, low-weight expert blocking | accuracy: 98.2% to 15.1%; 86.5% to 77.7% | Helped |
| remove expert dropout (ablation) | Mixture-of-Pathways · Mod-Cog tasks | Fig. 6b correlation: 0.31 -> 0.03 | Hurt |
| increase routing-cost coefficient to α=1e⁻⁴ (ablation) | Mixture-of-Pathways · Mod-Cog tasks | accuracy: 83.0% ± 15.5% -> 69.0% ± 20.1% | Hurt |
| increase routing-cost coefficient to α=1e⁻⁴ (ablation) | Mixture-of-Pathways · Mod-Cog tasks | Fig. 5 correlation: 0.54 -> -0.57 | Hurt |
| decrease routing-cost coefficient to α=1e⁻⁶ (ablation) | Mixture-of-Pathways · Mod-Cog tasks | Fig. 6b correlation: 0.31 -> 0.18 | Hurt |
| remove task embedding layer (ablation) | Mixture-of-Pathways · Mod-Cog tasks | Fig. 6b correlation: 0.31 -> 0.58 | Helped |
| change router hidden size to 32 or 128 (ablation) | Mixture-of-Pathways · Mod-Cog tasks | pathway correlations | No clear effect |
| use routing cost with task-performance scaling and expert dropout instead of baseline HMoE | Mod-Cog (82 tasks) · Mod-Cog (82 tasks) | accuracy: 91.1% ± 8.9% -> 83.0% ± 15.5% | Hurt |
| use routing cost with task-performance scaling and expert dropout instead of baseline HMoE | Mod-Cog tasks · Mod-Cog tasks | Fig. 5 correlation: -0.01 -> 0.54 | Helped |
Surprising findings
- Routing cost alone without dropout reproduces the difficulty-complexity correlation but not the learning-time pathway shift, indicating dropout is needed for the brain-like learning dynamics.
- Expert dropout slightly reduces routing consistency (0.71 to 0.51) while improving self-sufficiency.
Limitations
- Architecture is feed-forward and lacks the recurrent loop structures of the primate brain.
- Small-scale simple tasks and small models; scalability to larger networks untested.
- Router is not linked to a specific brain region; pathway identification relies on three independent tests.
Experimental setup
Datasets: Mod-Cog
Benchmarks: Mod-Cog
Metrics: task accuracy routing consistency (pairwise correlation of LPC) self-sufficiency accuracy after blocking low-weight experts correlation of LPC with task difficulty lesion accuracy
Baselines: Baseline HMoE model
Builds on: Heterogeneous Mixture-of-Experts NeuroGym Mod-Cog Schedule-Free AdamW
Closest papers by meaning
- Neuron-based Multifractal Analysis of Neuron Interaction Dynamics in Large Models (ICLR 2025 · similarity 0.85)
- A Local-Ascending-Global Learning Strategy for Brain-Computer Interface (AAAI 2024 · similarity 0.85)
- $\textit{NeuroPath}$: A Neural Pathway Transformer for Joining the Dots of Human Connectomes (NeurIPS 2024 · similarity 0.84)
- Flexible task abstractions emerge in linear networks with fast and bounded units (NeurIPS 2024 · similarity 0.84)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.84)
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.