Research map / Architectures and efficiency / Mixture of experts

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: MoE LLM compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NSFT, a fine-grained PEFT method that selects task-relevant channel-group sub-experts inside MoE experts and rescales learning rate and gradients to compensate for sparse partial updates.

Problem

Expert-level sparse fine-tuning of MoE LLMs updates whole experts, which is too coarse since only a small fraction of intermediate channels in activated experts respond strongly to downstream tasks.

Approach

Decompose each expert along its intermediate dimension into contiguous channel groups (sub-experts), score them by routing importance times intra-expert gate-activation saliency, and fine-tune only the top groups; add learning-rate scaling by inverse active ratio and dynamic, entropy-adaptive gradient masks to offset the reduced update magnitude.

Key results

• On OLMoE, NSFT-OLMoE reaches an in-domain average of 46.62 vs 43.42 for ESFT-OLMoE with about 5.67% average trainable parameters, while general average is 45.96 vs 46.24 for ESFT.
• On Ling-mini-2.0 TableQA, NSFT-Ling attains a PoT average of 34.44 with 2.86% trainable parameters, versus 17.68 for ESFT-Ling (3.35%).
• NSFT saves up to 66.9% trainable parameters versus LoRA-QKVO+MLP under comparable or stronger MMedC performance.

Surprising findings

• Expert-level granularity (G=1024 on Ling-mini) degrades TableQA PoT to 17.68, far below sub-expert settings, suggesting finer MoE backbones need finer adaptation units.
• Static gradient scaling can be absorbed by Adam, motivating dynamic scaling refresh during training.

Limitations

• Optimal group size and threshold are backbone- and task-dependent, requiring a validation sweep.
• On Ling-mini-2.0 general average, ESFT-Ling (67.03) slightly exceeds NSFT-Ling (66.18); on OLMoE general average NSFT is slightly below ESFT.

Experimental setup

Datasets: MMedC SciRIFF RAG-Instruct MATH GSM8K Code (HumanEval, MBPP) TableInstruct PubMedQA Health_Claims

Benchmarks: MMedC SciRIFF RAG-Instruct MATH Code TableQA GPQA MMLU-Redux C-Eval IFEval

Metrics: accuracy average score percentage of trainable parameters

Baselines: Full FT LoRA-r32 LoRA-r64 ESFT

Builds on: ESFT LoRA MoNE

Closest papers by meaning

• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.91)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
• FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.91)
• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.90)
• PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning (NeurIPS 2025 · similarity 0.90)
• RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism (NeurIPS 2026 · similarity 0.90)