Research map / Architectures and efficiency / Mixture of experts

DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models

CVPR 2025 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeRS (Decompose, Replace, Synthesis), which splits upcycled MoE experts into a shared base FFN plus lightweight expert-specific deltas, used both to compress trained upcycled MoE models (sparsification/quantization) and to train upcycled MoE with sparse or low-rank deltas.

Problem

Upcycled Mixture-of-Experts models duplicate dense FFN layers into many experts, causing heavy parameter overhead both during training and at inference.

Approach

Observe that upcycled experts stay extremely similar (cosine similarity >0.999) to the original FFN, so represent each expert as a shared base weight plus a compressed delta; compress deltas post-training (DeRS compression) or train only shared base plus sparse/low-rank deltas from the start (DeRS upcycling).

Key results

• DeRS-SM upcycling on MoE-LLaVA-Phi improves overall score from 60.8 to 61.1 while adding only 1.11M parameters versus 2.52B for vanilla upcycling (about 2270x reduction).
• DeRS compression with 90% sparsification of expert deltas (MoE-LLaVA-StableLM) keeps performance essentially unchanged, reducing MoE layer parameters by about 65%.
• On code generation, DeRS-SM and DeRS-LM upcycling raise overall score by 0.7-0.8 points (60.2 to 61.0/60.9) while cutting over 2B extra parameters; DeRS-LM reduces model size by 52.7% and inference memory by 43.8%.

Measured effects reported in the paper

Surprising findings

• Extreme compression of expert deltas (99% sparsity or 1-bit quantization) causes little degradation, attributed to the base FFN already carrying the fine-tuned knowledge.
• Freezing the shared base FFN in DeRS upcycling drops performance by 1.3-1.6 points, so the base must be trainable.

Limitations

• Sparse-matrix deltas can only modify a subset of elements of a same-shape matrix, so lower sparsity is needed for distinctiveness, increasing parameters.
• Experiments limited to upcycling from small dense models (1.3B-2.7B) and fine-tuning stage with limited training budgets.
• Compression gains depend on redundancy that is observed to vary across tasks (lower for code than medical).

Experimental setup

Datasets: LLaVA-mix-665k VQA-RAD SLAKE PathVQA evol-codealpaca-v1 MIMIC-IT LRV SViT LVIS

Benchmarks: VQAv2 GQA VisWiz ScienceQA-IMG TextVQA POPE MMBench MM-Vet VQA-RAD SLAKE PathVQA HumanEval

Metrics: accuracy pass@1 overall average score added parameters training memory inference memory

Baselines: Vanilla upcycling (MoE-LLaVA) Med-MoE Coder-MoE EEP MC-SMoE Expert sparsification (Lu et al.)

Builds on: sparse upcycling MoE-LLaVA Med-MoE Coder-MoE

Closest papers by meaning

• BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.91)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.91)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
• Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization (ICLR 2025 · similarity 0.90)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.90)
• Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.89)