Research map / Large language models / Instruction and fine-tuning
Ensembles of Low-Rank Expert Adapters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELREA, which clusters training instructions by gradient direction, trains LoRA expert adapters per cluster initialized from a base adapter, and routes test inputs via gradient similarity to ensemble them at inference.
Problem
Fine-tuning LLMs on diverse multi-source data causes conflicting gradient directions that hinder optimization and generalization; existing data selection methods are target-task dependent and need validation data.
Approach
Cluster training instructions by projected, normalized instruction-token gradient features (BIRCH), fine-tune one LoRA expert per cluster from a shared base adapter, and at inference weight the base and expert adapters' output logits by softmax-standardized cosine similarity between the test instruction gradient and cluster centroids.
Key results
- On MATH-Combined with Gemma-2b at LoRA rank 8, ELREA reaches 20.41% micro-average vs 18.61% for the base adapter (+1.80), and 27.33% vs 26.39% at rank 64 (+0.94).
- On BBH and MMLU with rank 8, ELREA gains +0.83 and +1.11 exact-match points over the base adapter (macro average 31.44 vs 30.47).
- With Gemma2-9b the gains narrow to +0.49 (rank 8) and +0.21 (rank 64) on MATH-Combined.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use gradient-based ELREA experts and input-dependent routing instead of the full-data LoRA adapter | Gemma-2b, LoRA rank 8 · MATH-Combined micro-average | accuracy (%): 18.61 -> 20.41 | Helped |
| use gradient-based ELREA experts and input-dependent routing instead of the full-data LoRA adapter | Gemma-2b, LoRA rank 8 · BBH and MMLU macro-average | accuracy (%): 30.47 -> 31.44 | Helped |
| use ELREA instead of the full-data adapter | Gemma2-9b, LoRA rank 8 · MATH-Combined micro-average | accuracy (%): 58.11 -> 58.60 | Helped |
| use ELREA instead of the full-data adapter | Gemma2-9b, LoRA rank 64 · MATH-Combined micro-average | accuracy (%): 61.17 -> 61.38 | Helped |
| add gradient-based clustering instead of randomly assigning training examples to clusters (ablation) | ELREA, Gemma-2b rank 8 · MATH-Combined micro-average | accuracy (%): 20.30 -> 20.41 | Helped |
| add gradient-based clustering instead of randomly assigning training examples to clusters (ablation) | ELREA, Gemma-2b rank 8 · BBH and MMLU macro-average | accuracy (%): 31.14 -> 31.44 | Helped |
| use input-dependent cluster routing instead of uniform cluster weights (ablation) | ELREA, Gemma-2b rank 8 · MATH-Combined micro-average | accuracy (%): 20.16 -> 20.41 | Helped |
| use input-dependent cluster routing instead of uniform cluster weights (ablation) | ELREA, Gemma-2b rank 8 · BBH and MMLU macro-average | accuracy (%): 30.83 -> 31.44 | Helped |
| replace BIRCH with 256-d PCA-based clustering (ablation) | ELREA, Gemma-2b rank 64 · MATH-Combined | accuracy (%): 27.33 -> 26.27 | Hurt |
| replace BIRCH clustering with K-means (ablation) | ELREA, Gemma-2b rank 64 · MATH-Combined | accuracy (%): 27.33 -> 27.00 | Hurt |
| use K-means without gradient normalization (ablation) | ELREA, Gemma-2b rank 64 · MATH-Combined | accuracy (%): 27.33 -> 26.51 | Hurt |
| reduce gradient projection dimensionality from 8192 to 512 (ablation) | ELREA · BBH | exact-match accuracy | Hurt |
Surprising findings
- Dataset-specific adapters trained only on each MATH-Combined subset do not consistently outperform the general base adapter trained on the full mix.
- Self-Consistency majority voting substantially underperforms the base adapter (-5.49 points at rank 8).
Limitations
- Inference cost is higher than MoE because C+1 adapters are activated simultaneously, increasing memory use.
- Gains diminish when the backbone is already strong or well adapted to the target task (Gemma2-9b).
- Hyperparameter tuning is preliminary and experiments are limited to smaller-scale models.
Experimental setup
Datasets: Flan V2 CoT Dolly-15k OpenAssistant Conversations MATH GSM8k MathQA SVAMP MMLU BIG-Bench Hard
Benchmarks: MATH-Combined (MATH, GSM8k, SVAMP, MathQA) MMLU BBH
Metrics: accuracy exact-match accuracy micro-average macro-average
Baselines: Base LoRA adapter (M+Q_base) Dataset-wise adapter MoE Routing MoE Merging MoLE LoRA Ensembles Self-Consistency Instruction Embedding Random Cluster Uniform Weights
Builds on: LESS LoRA Deep Ensembles MoLE Self-Consistency
Closest papers by meaning
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.90)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.89)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.88)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.88)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.88)
- RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.