Research map / Architectures and efficiency / Pruning and efficient tuning
Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers
Peter Cho-Ho Lam, Ziyi Wang, Zirui Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical study of whether prefix-tuning on a pretrained autoregressive Transformer can elicit arbitrary output sequences, based only on the abstract's framing of expressiveness of prefixes.
Problem
The expressiveness of prefixes in prefix-tuning for pretrained autoregressive Transformers remains theoretically unclear; open question of whether a prefix can steer a frozen model to produce any target output sequence.
Approach
Frames prefix-tuning expressiveness as a question about whether a pretrained autoregressive Transformer can be steered by a prefix to produce arbitrary output sequences; the abstract does not state the specific construction or result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| construct prefix by attention matching with linear systems (training-free) | GPT-2 · controlling output tokens across source/target/user inputs | success rate: close to 1 for short lengths; 0.54-0.78 for longer settings | Improved |
| increase output length (ablation) | attention-matching prefix on GPT-2 · control success rate | success rate: 'and' InLen 100: 0.9653 (OutLen 1) -> 0.6710 (OutLen 10) | Hurt |
| increase input length (ablation) | attention-matching prefix on GPT-2 · control success rate | success rate | Mixed |
| raise attention limit from very small values to about 20-30 (ablation) | attention-matching prefix on GPT-2 · success rate over 2,450 trials | success rate | Improved |
| increase attention limit beyond about 20-30 (ablation) | attention-matching prefix on GPT-2 · success rate over 2,450 trials | success rate | No clear effect |
| increase prefix repetition count (ablation) | attention-matching prefix on GPT-2 · success rate over 2,450 trials | success rate | Improved |
| increase repetition from r=1 to r=10 (ablation) | attention-matching prefix on Qwen3-4B · Qwen3-4B control success | success rate: 0.9966 -> 1.0000 | Improved |
| use attention matching prefix instead of long system prompt | Qwen3-4B · steering to target output | success rate: 0.8462 -> 0.9966 | Improved |
| prepend repeated target output as prompt | Qwen3-4B · steering to target output | success rate: 0.0351 -> 0.1700 | No clear effect |
| mismatch source and user input lengths (ablation) | GPT-2 attack example · proof-of-concept attack | success rate: 1.00 -> 0.92 at InLen 100 | No clear effect |
| mismatch final token of source and user input (ablation) | GPT-2 attack example · proof-of-concept attack | success | Hurt |
Limitations
- Full text unavailable; only the abstract was provided, so results and constructions are unknown.
Experimental setup
Builds on: prefix-tuning
Closest papers by meaning
- Prompting a Pretrained Transformer Can Be a Universal Approximator (ICML 2024 · similarity 0.89)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.86)
- Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity (ICLR 2025 · similarity 0.86)
- How do Transformers Perform In-Context Autoregressive Learning ? (ICML 2024 · similarity 0.85)
- WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.85)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
