Research map / Architectures and efficiency / Pruning and efficient tuning
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a separate side network with ladder shortcuts instead of training parameters inside the backbone | T5-base · GLUE | Train memory (GB): 17.6 -> 5.5 | Helped |
| use a separate side network with ladder shortcuts instead of Adapter modules | T5-base · GLUE | Train memory (GB): 13.0 -> 5.5 | Helped |
| scale the backbone from T5-base to T5-large with LST | LST · GLUE | Avg. score: 84.1 -> 87.1 | Helped |
| scale the backbone from T5-large to T5-3B with LST | LST · GLUE | Avg. score: 87.1 -> 88.1 | Helped |
| use LST rather than Y-tuning | BART-large · GLUE (except STS-B) | Avg. score: 76.9 -> 82.1 | Helped |
| use LST instead of full fine-tuning | CLIP-T5 · VQA, GQA, NLVR², MSCOCO | Train memory (GB): 36.2 -> 15.3 | Helped |
| use LST instead of full fine-tuning | CLIP-T5 · VQA, GQA, NLVR², MSCOCO | Avg. score: 77.5 -> 76.9 | Hurt |
| initialize side-network weights from a pruned backbone rather than randomly (ablation) | LST · GLUE | Avg. score | Helped |
| add intermediate ladder shortcuts rather than using no shortcuts (ablation) | LST · GLUE | Avg. score | Helped |
| add intermediate ladder shortcuts rather than only one output shortcut (ablation) | LST · GLUE | Avg. score | Helped |
| use Transformer blocks with gates instead of Adapter blocks with gates (ablation) | LST side network · GLUE | Avg. accuracy (%): 83.1 -> 83.8 | Helped |
| replace gates with cross-attention in Transformer side blocks (ablation) | LST side network · GLUE | Avg. accuracy (%): 83.8 -> 83.0 | Hurt |
Closest papers by meaning
- UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory (CVPR 2024 · similarity 0.88)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.87)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.86)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.86)
- PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · similarity 0.86)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.