Research map / Large language models / Instruction and fine-tuning
HyperTuning: Toward Adapting Large Language Models without Back-propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate task-specific prefix parameters from few-shot examples instead of using standard multi-task prefix tuning | T5-Large · P3 held-out tasks (dev) | accuracy (AVG): 49.7 -> 54.6 | Helped |
| Generate task-specific LoRA parameters from few-shot examples instead of standard multi-task LoRA tuning | T5-Large · P3 held-out tasks (dev) | accuracy (AVG): 45.5 -> 53.3 | Helped |
| Fine-tune the downstream model alongside the hypermodel | HyperT5-Prefix · P3 held-out tasks (dev), T5-Large | accuracy (AVG): 54.6 -> 56.0 | Helped |
| Fine-tune the downstream model alongside the hypermodel | HyperT5-LoRA · P3 held-out tasks (dev), T5-Large | accuracy (AVG): 53.3 -> 55.6 | Helped |
| Use a hypermodel-generated prefix initializer rather than random initialization (ablation) | Prefix-MLP fine-tuning · P3 held-out tasks | accuracy (AVG): 68.6 -> 75.2 | Helped |
| Use a hypermodel-generated LoRA initializer rather than random initialization (ablation) | LoRA fine-tuning · P3 held-out tasks | accuracy (AVG): 72.7 -> 75.0 | Helped |
| Use two positive demonstrations in addition to the task definition (ablation) | HyperT5-Prefix · Super-NaturalInstructions Test, T5-Large | ROUGE-L (AVG): 37.1 -> 43.5 | Helped |
| Use two positive demonstrations in addition to the task definition (ablation) | HyperT5-LoRA · Super-NaturalInstructions Test, T5-Large | ROUGE-L (AVG): 34.9 -> 42.0 | Helped |
| Use two positive demonstrations in addition to the task definition (ablation) | HyperT5-Prefix · Super-NaturalInstructions Test, T5-XL | ROUGE-L (AVG): 38.9 -> 48.6 | Helped |
| Use two positive demonstrations in addition to the task definition (ablation) | HyperT5-LoRA · Super-NaturalInstructions Test, T5-XL | ROUGE-L (AVG): 38.9 -> 45.0 | Helped |
| Add hyperpretraining before multi-task fine-tuning (ablation) | HyperT5-Prefix and HyperT5-LoRA · P3 held-out tasks | average score | Helped |
| Increase hyperpretraining beyond 50k steps (ablation) | HyperT5-LoRA · P3 held-out tasks | average score | Hurt |
Closest papers by meaning
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.89)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.89)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.89)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.89)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.