Research map / Architectures and efficiency / Pruning and efficient tuning
Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Customized Polytropon (C-Poly), a multi-task PEFT method that splits LoRA skill modules into task-common shared skills and one task-specific skill per task, with a jointly learned skill allocation matrix, evaluated on SuperGLUE and Super-NaturalInstructions.
Problem
Parameter-efficient fine-tuning is underexplored for multi-task learning, where negative transfer and the seesaw effect hurt performance when tasks share parameters.
Approach
Explicitly divides modular LoRA skills into a task-common skillset (routed by a learned task-by-skill allocation matrix) and a task-specific skillset (one exclusive skill per task, initialized as an identity/diagonal allocation), combining both per task, with Gumbel-sigmoid relaxation for the binary allocation.
Key results
- On FLAN-T5-Large SuperGLUE, C-Poly reaches 83.21 average accuracy vs 82.31 for MoE-LoRA/MHR and 81.85 for LoRA.
- On GLM-10B SuperGLUE, C-Poly reaches 62.26 average accuracy vs 56.99 for Poly and 52.05 for LoRA.
- On T5-Large with 100 SuperNI tasks, C-Poly reaches 49.34 Rouge-1 vs 44.24 for MHR and 34.16 for LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add task-specific skills alongside task-common skills | Poly-style multi-task PEFT · SuperGLUE | accuracy (AVG): 56.99 -> 62.26 | Helped |
| Add task-specific skills alongside task-common skills (ablation) | Poly-style multi-task PEFT · SuperNI (100 tasks), T5-Large | Rouge1: 43.04 -> 49.34 | Helped |
| Increase task-specific skills from 1 to 2 while reducing common skills from 3 to 2 (ablation) | C-Poly · SuperNI (100 tasks), T5-Large | Rouge1: 49.34 -> 48.79 | Hurt |
| Increase task-specific skills from 1 to 3 while reducing common skills from 3 to 1 (ablation) | C-Poly · SuperNI (100 tasks), T5-Large | Rouge1: 49.34 -> 47.46 | Hurt |
| Increase task-specific skills from 1 to 2 while reducing common skills from 3 to 2 (ablation) | C-Poly · SuperNI (100 tasks), FLAN-T5-Large | Rouge1: 68.69 -> 68.47 | Hurt |
| Increase task-specific skills from 1 to 3 while reducing common skills from 3 to 1 (ablation) | C-Poly · SuperNI (100 tasks), GLM-10B | Rouge1: 49.53 -> 46.85 | Hurt |
Surprising findings
- On FLAN-T5-Large the gains over baselines on SuperNI are small (e.g., Rouge-1 68.69 vs 68.84 for MHR), likely because FLAN training data overlaps with NI tasks.
Limitations
- Each task gets only one task-specific skill (B=1) in experiments, and the optimal A/B split is left to be identified per task and dataset.
- Training is one epoch with a single seed reported; no variance or significance testing is given.
- Gains on FLAN-T5-Large are small and attributed to possible overlap between FLAN data and NI tasks.
Experimental setup
Datasets: Super-NaturalInstructions SuperGLUE
Benchmarks: SuperGLUE Super-NaturalInstructions
Metrics: accuracy Rouge-1 Rouge-L Rouge-LSum
Baselines: LoRA MoE-LoRA Polytropon Multi-Head Routing Full fine-tuning
Builds on: Polytropon LoRA MoLoRA Multi-Head Routing Customized Gate Control (CGC)
Closest papers by meaning
- Selective Task Group Updates for Multi-Task Optimization (ICLR 2025 · similarity 0.85)
- Quantifying Task Priority for Multi-Task Optimization (CVPR 2024 · similarity 0.84)
- Towards Consistent Multi-Task Learning: Unlocking the Potential of Task-Specific Parameters (CVPR 2025 · similarity 0.84)
- DMTG: One-Shot Differentiable Multi-Task Grouping (ICML 2024 · similarity 0.84)
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.84)
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.