Research map / Large language models / Instruction and fine-tuning
From Instance Training to Instruction Learning: Task Adapters Generation from Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAGI, which uses a hypernetwork to turn task instructions into LoRA adapters for T5 models, trained with hypernetwork pretraining plus knowledge distillation (label, logit, and adapter-parameter alignment) against instance-trained LoRA teachers.
Problem
Instruction finetuning relies on instance-level training over large task data, which limits adaptation to unseen tasks with scarce labeled instances; existing hypernetwork methods that generate adapters from instructions lag behind meta-training.
Approach
Generate task-specific LoRA adapters from instructions with a hypernetwork that has a cross-attention instruction-fusion encoder, and train it by distilling from instance-trained task LoRA teachers via KL on outputs plus MSE on adapter parameters, after pretraining the hypernetwork on C4.
Key results
- On SNI XL (3B) with Def+2Pos, TAGI reaches 56.3 RougeL vs 54.0 for Tk-Instruct and 53.2 for HINT, at about 0.39x the relative FLOPs.
- On P3 with T0 at 3B, TAGI averages 58.8 on MTest11 vs 56.8 for T0 Metatrain and 58.2 for T0-FiD ICL, at 0.88x relative inference time.
- Ablation on T5-LM-XL shows removing instruction fusion drops SNI Def+2Pos from 56.3 to 40.6, and removing the KL loss drops it to 53.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hypernetwork pretraining (ablation) | TAGI · SNI Def | ROUGE-L: 47.1 -> 48.4 | Helped |
| Add instruction fusion (ablation) | TAGI · SNI Def | ROUGE-L: 35.1 -> 48.4 | Helped |
| Add cross-entropy prediction loss (ablation) | TAGI · SNI Def | ROUGE-L: 47.6 -> 48.4 | Helped |
| Add KL distillation loss (ablation) | TAGI · SNI Def | ROUGE-L: 45.7 -> 48.4 | Helped |
| Add adapter parameter alignment loss (ablation) | TAGI · SNI Def | ROUGE-L: 47.5 -> 48.4 | Helped |
| Add a hypernetwork (ablation) | TAGI · SNI Def | ROUGE-L: 43.8 -> 48.4 | Helped |
| Use LoRA rank 16 instead of rank 32 (ablation) | TAGI · SNI Def + 2 Pos. | ROUGE-L: 42.5 -> 41.8 | Hurt |
| Use LoRA rank 64 instead of rank 32 (ablation) | TAGI · SNI Def + 2 Pos. | ROUGE-L: 42.5 -> 42.3 | No clear effect |
| Increase the number of meta-training tasks (ablation) | TAGI and comparison methods · SNI and P3 | performance | Helped |
| Use T5-LM initialization instead of T0 | TAGI · P3, large or XL models | accuracy | Helped |
| Use TAGI instead of HINT | T5-LM XL (3B) · SNI Def | ROUGE-L: 47.2 -> 48.4 | Helped |
| Use TAGI instead of T0 metatraining | T0 XL (3B) · P3 MTest11 Avg. | accuracy: 56.8 -> 58.8 | Helped |
Surprising findings
- Initializing from T5-LM outperforms T0 for large and XL sizes, which the authors attribute to possible dilution of knowledge during T0 training.
- TAGI's 11B SNI result trails Tk-Instruct, which the authors attribute to insufficient training steps and omitted parameter alignment at that size.
Limitations
- Experiments are limited to encoder-decoder models up to 3B (11B only partially trained), so scaling to 7B+ or decoder-only models is untested.
- Training cost is substantially higher than baselines due to distillation, per-batch hypernetwork adapter generation, and pretraining teacher task models.
- SNI evaluation is restricted to English tasks; multilingual generalization is unexplored.
Experimental setup
Datasets: Super-Natural Instructions P3 C4
Benchmarks: Super-Natural Instructions P3 MTest11 HyperT5 evaluation
Metrics: ROUGE-L accuracy relative FLOPs relative inference time
Baselines: Tk-Instruct T0 HINT HyperTuning Hypter FiD-ICL Full FT GPT-2 XL OPT (13B)
Builds on: HINT HyperTuning LoRA Tk-Instruct T0
Closest papers by meaning
- #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models (ICLR 2024 · similarity 0.89)
- Instruction Embedding: Latent Representations of Instructions Towards Task Identification (NeurIPS 2024 · similarity 0.88)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- Hypothesis, Verification, and Induction: Grounding Large Language Models with Self-Driven Skill Learning (AAAI 2024 · similarity 0.88)
- Self-Alignment with Instruction Backtranslation (ICLR 2024 · similarity 0.88)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.