Research map / Vision-language and multimodal / CLIP and zero-shot
Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes zero-shot transfer of continuous soft prompts between masked language models by encoding source prompts as cosine similarities to shared-vocabulary anchors and searching target embeddings that match this relative representation, extended to multi-source transfer.
Problem
Continuous prompts tuned on one language model are not directly transferable to another with different embedding space and dimension; prior cross-model transfer needs a trained projector and task supervision on the target.
Approach
Encode a source continuous prompt as cosine similarities to a set of shared-vocabulary anchor embeddings (relative space), then zero-shot search target prompt embeddings whose relative representation under target anchors matches it, optionally summing matches over multiple source models.
Key results
- Single-source zero-shot transfer from BERT base to BERT large reaches 31.40% and from RoBERTa base to BERT base reaches 31.33%, surpassing manual prompts (30.64% on BERT base) and the neural projector baseline.
- Dual-source BERT base + RoBERTa base transfer improves ALBERT base to 27.13% and ALBERT large to 26.54%, 1-2 points above the best single source on those unseen targets.
- Direct transfer between same-dimension models is near zero (0.11-0.12% between BERT and RoBERTa base), showing raw continuous prompts are not transferable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use relative-space prompt transfer instead of direct transfer | continuous prompts transferred between BERT-base and RoBERTa-base · LAMA-TREx | accuracy: 0.11 -> 17.68; 0.12 -> 31.33 | Helped |
| Combine BERT-base and BERT-large as sources | single-source transfer from BERT-base · LAMA-TREx | accuracy: outperforms BERT_base by 2--10 percentage points | Mixed |
| Combine BERT-base and RoBERTa-base as sources | best single-source transfer · LAMA-TREx on ALBERT-base and ALBERT-large | accuracy: 1--2 points | Helped |
| Use prompts induced by a larger source model | single-source prompt transfer · LAMA-TREx | accuracy | Hurt |
| Apply target-embedding normalization (ablation) | transferred prompts when source and target models differ · LAMA-TREx | accuracy | Helped |
| Apply target-embedding normalization (ablation) | prompt transfer when source and target are identical · LAMA-TREx | accuracy | Hurt |
| Increase anchor count from 512 toward 8192 (ablation) | relative-space prompt transfer · LAMA-TREx | accuracy | Helped |
| Use the entire shared vocabulary as anchors (ablation) | relative-space prompt transfer · LAMA-TREx | accuracy | Hurt |
| Increase prompt length up to five virtual tokens (ablation) | relative-space prompt transfer · LAMA-TREx | accuracy | Helped |
| Increase prompt length to 10 virtual tokens (ablation) | relative-space prompt transfer · LAMA-TREx | accuracy | Hurt |
| Reduce relative-space matching loss during search | zero-shot prompt transfer · LAMA-TREx | validation accuracy | Helped |
| Transfer RoBERTa-base prompts rather than use manual prompts | RoBERTa-large · SST-2 | accuracy: 69.95 -> 84.63 | Helped |
Surprising findings
- Larger source models (BERT large, RoBERTa large) yield less transferable prompts than base models, which the authors attribute to model-specific information in large expressive embedding spaces.
- Normalization of searched target embeddings hurts self-transfer slightly but substantially improves cross-model transfer.
Limitations
- Transfer from large source models is weak, limiting the approach to small-source, large-target scenarios.
- Only masked language models are used as sources and targets in the main experiments; decoder GPT-2 sources underperform manual prompts.
- Evaluated mainly on factual probing (LAMA-TREx) with a limited set of encoder models.
Experimental setup
Datasets: LAMA-TREx SST-2 DBpedia
Benchmarks: LAMA-TREx SST-2 DBpedia
Metrics: micro-average accuracy
Baselines: Random Manual prompts Direct tuning Discretization Neural projector
Builds on: OptiPrompt relative representations prompt tuning
Closest papers by meaning
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.88)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.87)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.87)
- Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning (AAAI 2024 · similarity 0.86)
- Adaptive Prompt Routing for Arbitrary Text Style Transfer with Pre-trained Language Models (AAAI 2024 · similarity 0.86)
- One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.