Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Learning to Route Among Specialized Experts for Zero-Shot Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhatGoose, a post-hoc method that trains a sigmoid gate per frozen LoRA module on a pool of PEFT experts and routes each token per module via top-k, improving zero-shot generalization of T5-XL without access to expert training data.
Problem
Recycling large collections of PEFT-trained expert language models to improve zero-shot generalization to unseen tasks, without simultaneous access to the experts' training datasets and with little extra compute.
Approach
Freeze each trained LoRA module and train only a small sigmoid gate vector per module for 100 steps; at inference, route each token and each layer to the top-2 modules by cosine similarity between standardized gate vectors and activations, enabling per-token, per-module routing rather than picking a single expert per query.
Key results
- In the T0 Held-In setting, PhatGoose reaches 56.9 average on T0HO versus 51.6 for T0-3B multitask and 57.2 for oracle routing, and outperforms other post-hoc routing methods.
- On BBL in the T0 Held-In setting, PhatGoose scores 37.3 versus 36.6 for multitask T0-3B and 34.5 for Arrow.
- In the FLAN setting with 166 experts, PhatGoose outperforms all other routing methods on BBH (35.6) and BBL (35.2), but the gap to oracle routing is larger (45.5 and 46.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned post-hoc gates and tokenwise routing instead of retrieval routing | PHATGOOSE with the T0 Held-In expert pool · T0HO | accuracy: 50 -> 56.9 | Helped |
| Use learned post-hoc gates and tokenwise routing instead of retrieval routing | PHATGOOSE with the T0 Held-In expert pool · BBH | accuracy: 30.9 -> 34.9 | Helped |
| Use learned post-hoc gates and tokenwise routing instead of retrieval routing | PHATGOOSE with the T0 Held-In expert pool · BBL | accuracy: 33.6 -> 37.3 | Helped |
| Use learned post-hoc gates and tokenwise routing instead of retrieval routing | PHATGOOSE with the FLAN expert pool · BBH | accuracy: 31.4 -> 35.6 | Helped |
| Use learned post-hoc gates and tokenwise routing instead of retrieval routing | PHATGOOSE with the FLAN expert pool · BBL | accuracy: 33.1 -> 35.2 | Helped |
| Replace learned gate vectors with average dataset activations (ablation) | PHATGOOSE with the T0 Held-In expert pool · T0HO | accuracy: 56.9 -> 50.2 | Hurt |
| Replace learned gate vectors with average dataset activations (ablation) | PHATGOOSE with the T0 Held-In expert pool · BBH | accuracy: 34.9 -> 33.8 | Hurt |
| Replace learned gate vectors with average dataset activations (ablation) | PHATGOOSE with the FLAN expert pool · BBL | accuracy: 35.2 -> 34 | Hurt |
| Jointly train gate and module parameters instead of training gates post-hoc with the module frozen (ablation) | PHATGOOSE · T0 Held-Out | accuracy: 51.8 -> 46.2 | Hurt |
Surprising findings
- Routing alignment with the oracle router correlates weakly with performance (Pearson -0.2), suggesting different routing strategies can be effective.
- On CB, PhatGoose nearly never routes to the oracle module yet outperforms oracle routing by about 10%.
Limitations
- Only evaluated on T5-family encoder-decoder models; decoder-only LLMs not tested.
- Gains do not consistently increase as the expert pool grows; BBL performance drops with more experts in the FLAN setting.
- Requires an extra gate-training step per contributor, and the FLAN multitask comparison model is not a fair comparison.
Experimental setup
Datasets: T0 Held-In prompted datasets (P3/PromptSource) FLAN Collection zero-shot prompted datasets
Benchmarks: T0 Held-Out (T0HO) BIG-bench Hard (BBH) BIG-bench Lite (BBL)
Metrics: accuracy
Baselines: Multitask (T0-3B / FLAN-T5-XL) Retrieval Arrow Merged Experts Average Activation Oracle Best Individual
Builds on: LoRA T0 FLAN Arrow routing LoRAHub
Closest papers by meaning
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.90)
- Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.87)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.86)
- Agent Instructs Large Language Models to be General Zero-Shot Reasoners (ICML 2024 · similarity 0.86)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.86)
- A Generalization Theory for Zero-Shot Prediction (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.