Research map / Architectures and efficiency / Pruning and efficient tuning
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| T-Few PEFT instead of few-shot ICL with GPT-3 175B | few-shot learning · T0 held-out tasks | accuracy: 66.6% -> 72.4% | Helped |
| T-Few instead of ICL | few-shot learning · T0 held-out tasks | inference FLOPs: 1.4e15 -> 1.1e12 | Helped |
| T-Few on RAFT | T0 · RAFT | accuracy: 69.6% -> 75.8% | Helped |
| add length-normalization loss (ablation) | full fine-tuning T0-3B · T0 held-out tasks | accuracy: 60.7 -> 62.71 | Helped |
| add unlikelihood loss on top of LN (ablation) | full fine-tuning T0-3B · T0 held-out tasks | accuracy: 62.71 -> 63.3 | Helped |
| (IA)^3 instead of full fine-tuning | T0-3B · T0 held-out tasks | accuracy | Helped |
| prompt tuning instead of full fine-tuning | T0-3B · T0 held-out tasks | accuracy | Hurt |
| pre-train (IA)^3 parameters on multitask mixture (ablation) | (IA)^3 T0-3B · T0 held-out tasks | accuracy: 64.6 -> 65.8 | Helped |
| add (IA)^3 pre-training (ablation) | T-Few on T0 · T0 held-out tasks | accuracy: +1.6% | Helped |
| add UL and LN losses (ablation) | T-Few on T0 · T0 held-out tasks | accuracy: +4.1% | Helped |
| few-shot ICL instead of zero-shot for T0 | T0 · T0 held-out tasks | accuracy | Hurt |
Closest papers by meaning
- Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.91)
- The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.90)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.89)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.88)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.