AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers

Peter Cho-Ho Lam, Ziyi Wang, Zirui Zhou

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretical study of whether prefix-tuning on a pretrained autoregressive Transformer can elicit arbitrary output sequences, based only on the abstract's framing of expressiveness of prefixes.

Problem

The expressiveness of prefixes in prefix-tuning for pretrained autoregressive Transformers remains theoretically unclear; open question of whether a prefix can steer a frozen model to produce any target output sequence.

Approach

Frames prefix-tuning expressiveness as a question about whether a pretrained autoregressive Transformer can be steered by a prefix to produce arbitrary output sequences; the abstract does not state the specific construction or result.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
construct prefix by attention matching with linear systems (training-free)GPT-2 · controlling output tokens across source/target/user inputssuccess rate: close to 1 for short lengths; 0.54-0.78 for longer settingsImproved
increase output length (ablation)attention-matching prefix on GPT-2 · control success ratesuccess rate: 'and' InLen 100: 0.9653 (OutLen 1) -> 0.6710 (OutLen 10)Hurt
increase input length (ablation)attention-matching prefix on GPT-2 · control success ratesuccess rateMixed
raise attention limit from very small values to about 20-30 (ablation)attention-matching prefix on GPT-2 · success rate over 2,450 trialssuccess rateImproved
increase attention limit beyond about 20-30 (ablation)attention-matching prefix on GPT-2 · success rate over 2,450 trialssuccess rateNo clear effect
increase prefix repetition count (ablation)attention-matching prefix on GPT-2 · success rate over 2,450 trialssuccess rateImproved
increase repetition from r=1 to r=10 (ablation)attention-matching prefix on Qwen3-4B · Qwen3-4B control successsuccess rate: 0.9966 -> 1.0000Improved
use attention matching prefix instead of long system promptQwen3-4B · steering to target outputsuccess rate: 0.8462 -> 0.9966Improved
prepend repeated target output as promptQwen3-4B · steering to target outputsuccess rate: 0.0351 -> 0.1700No clear effect
mismatch source and user input lengths (ablation)GPT-2 attack example · proof-of-concept attacksuccess rate: 1.00 -> 0.92 at InLen 100No clear effect
mismatch final token of source and user input (ablation)GPT-2 attack example · proof-of-concept attacksuccessHurt

Limitations

Experimental setup

Builds on: prefix-tuning

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy