Research map / Architectures and efficiency / Pruning and efficient tuning
MoLEx: Mixture of Layer Experts for Fine-tuning with Sparse Upcycling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoLEx, a sparse-upcycling mixture-of-layer-experts that mixes pre-trained layers as experts on top of PEFT (LoRA) for better fine-tuning on GLUE and E2E with negligible overhead.
Problem
Parameter-efficient fine-tuning of large pre-trained language models leaves unexploited the complementary linguistic information across layers; the goal is to improve PEFT accuracy and transfer without adding effective parameters.
Approach
Treat the pre-trained layers themselves as experts in a sparse MoE (top-1 routing with a shared gate) applied at every layer, mixing the original layer with a routed layer via a learnable/fixed weight alpha, so the model reuses existing weights and only adds a small router.
Key results
- On GLUE with RoBERTa-base, MoLEx (LoRA + MoLEx) averages 86.3 vs 85.7 for LoRA with 0.309M vs 0.3M parameters.
- On E2E with GPT-2 medium, MoLEx raises BLEU from 70.0 (LoRA) to 70.7 with a 0.359M parameter count.
- Zero-shot transfer from QQP to MRPC improves from 67.2 (LoRA) to 69.9 (MoLEx) accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MoLEx layer mixing to LoRA | RoBERTa-base · GLUE tasks | average score: 85.7 -> 86.3 | Helped |
| Add MoLEx layer mixing to LoRA | RoBERTa-large · GLUE tasks | average score: 88.8 -> 89.1 | Helped |
| Add MoLEx layer mixing to LoRA | GPT-2 medium · E2E NLG | BLEU: 70.0 ±.5 -> 70.7 ±.4 | Helped |
| Add MoLEx layer mixing to LoRA | GPT-2 medium · E2E NLG | METEOR: 46.8 ±.2 -> 46.5 ±.09 | Hurt |
| Add MoLEx layer mixing to LoRA | GPT-2 medium · E2E NLG | CIDEr: 2.52 ±.01 -> 2.52 ±.01 | No clear effect |
| Add MoLEx layer mixing to LoRA | RoBERTa-base · GLUE QNLI | accuracy: 93.2 ±.2 -> 93.2 ±.2 | No clear effect |
| Add MoLEx layer mixing to LoRA | RoBERTa-base · GLUE MNLI checkpoint evaluation on RTE | accuracy: 86.8 ±.2 -> 86.8 ±.2 | No clear effect |
| Add MoLEx layer mixing to LoRA | RoBERTa-base · GLUE QNLI with random input noise | accuracy: 63.1 ±.2 -> 64.0 ±.2 | Helped |
| Add MoLEx layer mixing to LoRA | RoBERTa-base · GLUE SST-2 with random input noise | accuracy: 69.3 ±.1 -> 70.9 ±.2 | Helped |
| Use Top1 rather than Top2 routing (ablation) | MoLEx on RoBERTa-base · GLUE CoLA | accuracy: 63.7 ±.4 -> 64.8 ±.5 | Helped |
| Use Top1 rather than Top2 routing (ablation) | MoLEx on RoBERTa-base · GLUE QQP | accuracy: 90.7 ±.0 -> 91.0 ±.0 | Helped |
| Use Top1 rather than Top2 routing (ablation) | MoLEx on RoBERTa-base · GLUE SST-2 | accuracy: 95.0 ±.3 -> 95.4 ±.2 | Helped |
Surprising findings
- Top-1 routing outperforms top-2 routing in the ablation on CoLA, QQP and SST-2.
- Layer 11 of RoBERTa-base performs best on most probing tasks, unlike the pattern reported for BERT.
Limitations
- Robustness guarantee holds only for deep linear models; the nonlinear case is supported empirically only.
- Gains on several GLUE tasks are small, e.g., QNLI and MNLI-checkpoint results are equal to or near LoRA.
- Hyperparameters of the gate (type, projection dim, aggregation, load balance) are tuned per task.
Experimental setup
Datasets: GLUE E2E NLG Challenge SentEval probing tasks
Benchmarks: GLUE E2E NLG Challenge SentEval probing tasks
Metrics: accuracy Matthews correlation Pearson correlation BLEU NIST METEOR ROUGE-L CIDEr
Baselines: LoRA full fine-tuning BitFit Adapter (H) Adapter (L) Adapter (D) PreLayer FT Top2
Builds on: LoRA sparse upcycling sparse mixture of experts SentEval
Closest papers by meaning
- FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.88)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.88)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.87)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.