Research map / Architectures and efficiency / Mixture of experts
S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S'MoRE, a PEFT method that stacks hierarchical low-rank residual experts into a multi-layer GNN-like structure with top-down tree routing, matching LoRA's parameter count while improving accuracy over MoE-LoRA baselines.
Problem
Parameter-efficient fine-tuning of LLMs trades off parameter efficiency against capacity: LoRA is efficient but limited, while MoE-LoRA variants add under-utilized parameters and face routing scaling issues.
Approach
Decompose expert weights into hierarchical low-rank residuals arranged in a multi-layer structure; a top-down router selects a residual tree per token, and inter-layer aggregation (a GNN-like, GIN-style propagation) lets the same experts form exponentially many non-isomorphic structures without adding experts.
Key results
- On LLaMA 3.2-1B with dense gate, S'MoRE (2-2) reaches 61.30 average accuracy vs. 59.15 for LoRA and 59.93 for MixLoRA (4), with ~0.048B average params.
- On LLaMA 3-8B with dense gate, S'MoRE (2-2) averages 86.83 vs. 84.56 for LoRA across commonsense benchmarks.
- On Gemma 2-9B, S'MoRE (4-4) averages 77.35 across ARC-e, CSQA, Winogrande and HumanEval vs. 74.00 for LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace a flat MixLoRA layer with a two-layer S'MoRE structure | LLaMA 3.2 1B with dense gating · ARC-c/e, CSQA, OBQA, Winogrande (average) | Avg Acc.: 60.25 -> 61.30 | Helped |
| replace a flat MixLoRA layer with a two-layer S'MoRE structure | LLaMA 3 8B with dense gating · ARC-c/e, CSQA, OBQA, Winogrande (average) | Avg Acc.: 85.82 -> 86.83 | Helped |
| replace a flat MixLoRA layer with a two-layer S'MoRE structure | Gemma 2 9B · ARC-e, CSQA, Winogrande, HumanEval (average) | Avg Acc. / Pass@1: 75.86 -> 76.82 | Helped |
| add a third layer to the 2-2 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · Winogrande | Accuracy: 0.87 | Helped |
| add a third layer to the 2-2 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · ARC-c | Accuracy: 40.00 -> 39.32 | Hurt |
| add a third layer to the 2-2 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · ARC-e | Accuracy: 75.31 -> 74.25 | Hurt |
| add a third layer to the 2-2 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · Commonsense QA | Accuracy: 66.99 -> 67.40 | Helped |
| add a third layer to the 2-2 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · OpenBook QA | Accuracy: 72.20 -> 72.60 | Helped |
| add a third layer to the 4-4 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · ARC-c | Accuracy: 39.66 -> 40.34 | Helped |
| add a third layer to the 4-4 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · ARC-e | Accuracy: 74.43 -> 73.90 | Hurt |
| add a third layer to the 4-4 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · Commonsense QA | Accuracy: 67.32 -> 67.32 | No clear effect |
| add a third layer to the 4-4 configuration (ablation) | S'MoRE on LLaMA 3.2 1B · OpenBook QA | Accuracy: 72.80 -> 73.60 | Helped |
Surprising findings
- HydraLoRA and 1-layer MixLoRA do not improve accuracy with more parameters, whereas S'MoRE keeps scaling in the 0.4B-1B range on GSM8K/HumanEval.
Limitations
- Bottom-up router variant not evaluated due to limited GPU resources.
- Theoretical structural flexibility assumes binary router masks and an ideal router.
Experimental setup
Datasets: ARC-c ARC-e CommonsenseQA OpenBookQA Winogrande GSM8K CodeAlpaca
Benchmarks: ARC-c ARC-e CommonsenseQA OpenBookQA Winogrande GSM8K HumanEval
Metrics: accuracy Pass@1 trainable parameters (B)
Baselines: LoRA MixLoRA HydraLoRA
Builds on: LoRA MixLoRA HydraLoRA Graph Isomorphism Network noisy top-k gating
Closest papers by meaning
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.89)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.89)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.89)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism (NeurIPS 2026 · similarity 0.88)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.