Research map / Large language models / Instruction and fine-tuning
MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoRAgent, a parameter-efficient agent tuning method that splits agent capabilities into reasoner, executor, and summarizer roles, each with its own group of LoRAs with routers, trained on a GPT-4o/DeepSeek-V3-curated multi-role dataset.
Problem
Parameter-efficient fine-tuning of LLMs for agent tasks (function calling and tool use) underperforms full fine-tuning because one low-rank adapter must learn several distinct capabilities at once.
Approach
Decompose the Reason+Action agent capability into reasoner, executor, and summarizer roles; equip each with a group of LoRAs (shared plus Top-K token-routed), activate only one role per token via a rule-based role-aware gate, and train with cross-entropy plus load-balance and orthogonality losses.
Key results
- On StableToolBench DFS, MoRAgent-Llama (1.24B+0.16B) raises average pass rate by 44.5 points over its base model (10.0 to 54.5).
- On BFCL, MoRAgent-Phi (3.82B+0.36B) reaches 82.1 average accuracy versus 64.2 for base, a +17.9 gain, with 0.36B trainable parameters.
- On the BFCL ablation with Llama3.2-1B, MoRAgent (0.16B trainable) scores 77.6 vs 76.0 for full SFT (1.24B) and 65.5 for LoRA (0.16B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MoRAgent instead of the base model | Llama3.2-1B-Instruct · StableToolBench DFS | Pass rate: 10.0 -> 54.5 | Helped |
| apply MoRAgent instead of the base model | Phi3.5-mini-Instruct · StableToolBench DFS | Pass rate: 44.5 -> 58.7 | Helped |
| apply MoRAgent instead of the base model | Llama3.2-1B-Instruct · BFCL | AVG accuracy: 27.5 -> 77.6 (+50.1) | Helped |
| apply MoRAgent instead of the base model | Qwen2.5-1.5B-Coder · GSM8K | Accuracy: 54.7 -> 68.5 (+13.8) | Helped |
| apply MoRAgent instead of the base model | Qwen2.5-1.5B-Coder · MATH | Accuracy: 23.5 -> 45.5 (+12.0) | Helped |
| add balance loss (ablation) | MoRAgent · BFCL | AVG accuracy: 72.3 -> 75.1 | Helped |
| add orthogonal loss (ablation) | MoRAgent · BFCL | AVG accuracy: 72.3 -> 74.4 | Helped |
| add balance and orthogonal losses (ablation) | MoRAgent · BFCL | AVG accuracy: +5.3% | Helped |
| increase total LoRAs from 4/4/3 to 5/5/4 (ablation) | MoRAgent reasoner/executor/summarizer · BFCL | AVG accuracy: 73.2 -> 77.6 | Helped |
| increase total LoRAs from 5/5/4 to 6/6/5 (ablation) | MoRAgent reasoner/executor/summarizer · BFCL | AVG accuracy: 77.6 -> 79.1 | Helped |
| increase total LoRAs from 6/6/5 to 7/7/6 (ablation) | MoRAgent reasoner/executor/summarizer · BFCL | AVG accuracy: 79.1 -> 80.0 | Helped |
| increase fine-tuning samples from 1K to 90K (ablation) | MoRAgent · BFCL | AVG accuracy: 54.9 (+27.4) -> 77.6 (+50.1) | Helped |
Surprising findings
- Full-parameter SFT is outperformed by the proposed PEFT method (77.6 vs 76.0 on BFCL ablation) despite using far more trainable parameters.
- The 1B model MoRAgent-Llama surpasses GPT-4 on BFCL-style averages is not claimed; instead MoRAgent-Phi is reported to surpass ToolLLaMA-v2 with fewer parameters.
Limitations
- Accuracy gains depend on GPT-4o-completed and manually corrected multi-role data, which is costly to reproduce.
- Number of LoRAs per role was chosen by toy experiments and the paper notes performance grows with parameter count, so the efficiency-accuracy tradeoff is tuned.
- Evaluation on the MATH and GSM8K baselines is mixed, with some baseline numbers reported without clear settings.
Experimental setup
Datasets: ToolBench APIGen ToolACE glaive-function-calling-v2 MathGenie
Benchmarks: StableToolBench BFCL GSM8K MATH
Metrics: pass rate win rate AST accuracy execution accuracy relevance accuracy accuracy
Baselines: LoRA DoRA SFT ToolLLaMA-v2 GPT4 ToolACE-8B MiniCPM3-4B Llama3.2-3B-it Qwen2.5-72B-it DeepSeekMath-7B
Builds on: LoRA ReAct Switch Transformer ToRA MoLE
Closest papers by meaning
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.89)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.88)
- AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation (ICML 2026 · similarity 0.88)
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.88)
- TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture (ICLR 2026 · similarity 0.88)
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.