Research map / Architectures and efficiency / Pruning and efficient tuning
Universality and Limitations of Prompt Tuning
Yihan Wang, Jatin Chauhan, Wei Wang, Cho-Jui Hsieh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase soft prompt length beyond a threshold | prompt tuning, single-layer transformer · constructed memorization datasets S1-S3 | MSE loss | No clear effect |
| fine-tune the MLP layer instead of prompt tuning | single-layer transformer · constructed memorization datasets S1-S3 | MSE loss | Improved |
| use LoRA with width 2 on the MLP layer instead of prompt tuning | single-layer transformer · constructed memorization datasets S1-S3 | training MSE: less than 10^{-10} | Improved |
| train prompt longer until loss saturates | prompt tuning LLaMA 7B · SuperGlue RTE | soft prompt spectral norm | Mixed |
| train prompt longer until loss saturates | prompt tuning LLaMA 7B · WMT14 En-Fr | soft prompt spectral norm | Mixed |
Closest papers by meaning
- Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency (ICLR 2025 · similarity 0.94)
- Prompting a Pretrained Transformer Can Be a Universal Approximator (ICML 2024 · similarity 0.91)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.87)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.87)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.87)
- Ask, and it shall be given: On the Turing completeness of prompting (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
