Research map / Architectures and efficiency / Mixture of experts

dMoE: dLLMs with Learnable Block Experts

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes dMoE, a block-level expert routing method for MoE diffusion LLMs that aggregates token router scores per block into a top-p expert coreset, cutting unique activated experts ~4x with near-lossless accuracy on LLaDA2.0-mini.

Problem

In MoE diffusion LLMs, block parallel decoding activates many unique experts per forward pass because token-level routing selects experts independently per token, making MoE inference memory-bound.

Approach

Aggregate token-level router scores within each decoding block into block-level expert scores, select an adaptive top-p expert coreset, and restrict token-level routing to that coreset; train with self-distillation so training matches inference routing.

Key results

• Average unique expert count reduced from 69.5 to 16.575 per layer while retaining 99.55% of original average accuracy (83.95% to 83.575%).
• Memory usage reduced by 74.36% to 76.78% and end-to-end latency speedup of 1.36x to 2.04x (main text reports 1.14x to 1.66x on four benchmarks).
• At similar compression levels, DES-S* and DES-V* suffer large accuracy drops (e.g., MATH500 53.8% and 53.0% vs 71.2% for dMoE at p=0.6).

Surprising findings

• Reported speedup ranges are inconsistent between the abstract (1.36x-2.04x) and main results section (1.14x-1.66x).
• Top-4 baseline, which halves experts per token, drops MATH500 accuracy from 72.0% to 52.6%.

Limitations

• Evaluated only on LLaDA2.0-mini and language reasoning benchmarks; other modalities and MoE dLLMs untested.
• Does not reduce per-token computation; the number of experts per token stays fixed at 8.
• Official DES code unavailable, so DES was reimplemented by the authors.

Experimental setup

Datasets: GSM8K training set PRM12K Numina-Math OpenThoughts

Benchmarks: MATH500 GSM8K ARC-C MMLU (math subset)

Metrics: accuracy unique expert count memory footprint end-to-end latency

Baselines: Top-4 DES-S DES-V DES-S* DES-V*

Builds on: LLaDA2.0-mini block diffusion decoding DES self-distillation

Closest papers by meaning

• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.90)
• Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
• $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.89)
• LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.89)
• OLMoE: Open Mixture-of-Experts Language Models (ICLR 2025 · similarity 0.88)
• Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.88)