Research map / Architectures and efficiency / Mixture of experts

RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: MoE LLM compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RotMoLE, a MoE-LoRA variant whose extra rotation gate applies a learned low-rank rotation to each selected expert in addition to scalar gating, evaluated on multi-task and multilingual joint fine-tuning.

Problem

Conventional MoE-LoRA gating applies only scalar reweighting to selected experts, limiting expressiveness and expert specialization when experts are few and shared across diverse tasks or languages.

Approach

Add a rotation gate that outputs per-expert rotation angles and a learnable center vector to build an r x r rotation inside each low-rank expert (between A and B), so each selected expert is rotated as well as scaled, with only about dn to (d+r)n extra parameters.

Key results

• On mixed QA tasks with Llama-3.2-3B, RotMoLE beats MoE-LoRA in 7 of 8 settings, with average gains of roughly 1% to 10% (e.g., 42.62 vs 39.93 at r=2, n=2, k=2).
• On mixed GLUE tasks (Llama-3.2-3B, r=4, n=5, k=3), RotMoLE averages 73.42 vs 73.19 for MoE-LoRA and beats a size-equivalent MLP-gate variant (72.30).
• On mixed QA with Qwen-2.5-3B at r=6, n=4, k=2, RotMoLE averages 45.82 vs 41.42 for AdaMoLE, the best of the MoE-LoRA variants compared.

Surprising findings

• A size-equivalent MLP gate performs worse than the linear-gate MoE-LoRA baseline on GLUE, attributed to overfitting to the easiest sub-task.
• RotMoLE does not converge more slowly despite having more trainable parameters, and in some cases converges faster.

Limitations

• Evaluation limited to supervised fine-tuning; RL-based training such as GRPO not explored.
• Only adapters on the first quarter of layers are trained, to save compute, so generality to full-depth adaptation is untested.
• Rotation bias distribution analysis is qualitative; relation to task similarity not analyzed.

Experimental setup

Datasets: CommonsenseQA OpenBookQA SocialIQA SST-2 MRPC QNLI MTG title generation XNLI

Benchmarks: CommonsenseQA OpenBookQA SocialIQA GLUE (SST-2, MRPC, QNLI) MTG title generation XNLI

Metrics: accuracy Rouge-1 Rouge-2 Rouge-L relative standard deviation across tasks

Baselines: MoE-LoRA MoCLE MoLA AdaMoLE MoE-LoRA with MLP gate

Builds on: LoRA MoE-LoRA sparse mixture of experts LoRAMoE

Closest papers by meaning

• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
• From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.90)
• FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.89)
• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.89)
• LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.89)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)