AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

cMoLLM at Scale: Horizontal Scaling Laws for Convolutionally-Gated Mixture-of-LLMs

Xin Yang, Yemin Wang, Mingda Liu, Letian Li, Shuaishuai Cao, ZhengXiao He, Ryan Dong

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace dense FFN with cMoLLM streams (multi-head, 8 streams)GPT-2 85M · FineWeb validationperplexity: 25.40 -> 18.82Improved
increase the number of streams from 1 to 8 (multi-head gating) (ablation)cMoLLM multi-head · FineWeb validation, 85Mperplexity: 20.93 -> 18.82Improved
increase simple-gating streams from 4 to 8 (ablation)cMoLLM simple · FineWeb validation, 85Mperplexity: 19.53 -> 20.24Hurt
use context-aware gating instead of simple gating at 8 streams (ablation)cMoLLM · FineWeb GLUE, 85MGLUE accuracy (%): 44.68 -> 48.98Improved
multi-head cMoLLM vs dense baseline at 350MGPT-2 350M · FineWeb validationperplexity: 16.80 -> 14.03Improved
multi-head cMoLLM vs dense baseline at 760MGPT-2 760M · FineWeb validationperplexity: 12.79 -> 9.41Improved
multi-head cMoLLM vs dense baseline at 350MGPT-2 350M · GLUEGLUE accuracy (%): 50.68 -> 53.82Improved
multi-head cMoLLM vs dense baseline at 760MGPT-2 760M · SQuAD v2SQuAD v2 (%): 62.91 -> 64.56Improved
single stream (n=1) gated convolution vs dense FFN (ablation)GPT-2 85M · FineWeb validationperplexity: 25.40 -> 20.94Improved
increase streams from 1 to 8 with simple gating on SQuAD v2 (ablation)cMoLLM simple · SQuAD v2, 85MSQuAD v2 (%): 51.13 -> 55.95Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy