Research map / Label-efficient and robust learning / Domain adaptation and generalization
Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "Concentration," a lookback-attention objective (strength and fluctuation of attention on prompt tokens in deep layers) that is added to soft prompt tuning losses and hard prompt filtering/MARL matching to improve multi-source domain generalization of prompts.
Problem
Optimized prompts for pre-trained language models generalize poorly to unseen target domains, and existing remedies rely on target-domain data or costly pre-training, and do not apply to hard prompts.
Approach
Identifies that prompts receiving higher and more stable lookback attention from deep PLM layers generalize better, and turns this into concentration-reweighting losses for soft prompts and a concentration-based filter metric and agent reward (with multi-agent PPO matching) for hard prompts.
Key results
- Concentration-reweighted soft prompt tuning (both losses) raises average multi-source DG accuracy by 1.47% (Prompt Tuning), 1.78% (Prefix Tuning) and 1.02% (P-Tuning v2) over the vanilla methods on sentiment and NLI.
- Applying the hard-prompt framework to DP2O improves average accuracy by 1.34% on sentiment and 0.85% on NLI relative to vanilla DP2O.
- On Llama-2-7b-chat, Vicuna-7b-v1.5 and Alpaca-7b-wdiff, the method yields gains of roughly 1.4 to 2.1 points on MCQA (ROC, SCT, COPA) for Prefix Tuning and In-Context Demo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add concentration strength and concentration fluctuation losses to Prompt Tuning | Prompt Tuning · Multi-source domain generalization: sentiment and NLI settings | accuracy: 64.73 -> 66.19; 65.51 -> 69.54; 65.12 -> 65.89; 41.20 -> 42.48; 49.83 -> 50.31; 49.66 -> 50.42 | Helped |
| Add concentration strength loss only (ablation) | Prompt Tuning · Q+R→W | accuracy: 41.20 -> 41.53 | Helped |
| Add concentration fluctuation loss only (ablation) | Prompt Tuning · Q+R→W | accuracy: 41.20 -> 42.17 | Helped |
| Add either concentration loss alone instead of both losses (ablation) | Soft prompt methods · QNLI target-domain setting | accuracy | Hurt |
| Add GCS filtering and concentration-rewarded MARL | DP₂O · Multi-source domain generalization: sentiment and NLI settings | accuracy: 89.06 -> 89.63; 90.75 -> 92.87; 86.53 -> 87.85; 54.84 -> 56.42; 54.85 -> 55.32; 59.78 -> 61.27 | Helped |
| Add MARL prompt matching only (ablation) | DP₂O · S+M→C | accuracy: 89.06 -> 87.36 | Hurt |
| Add concentration-based filtering metric only (ablation) | DP₂O · S+M→C | accuracy: 89.06 -> 86.79 | Hurt |
| Add both concentration losses to Prompt Tuning | Prompt Tuning · In-domain SST-2 | accuracy: 73.84 -> 72.61 | No clear effect |
| Add both concentration losses to Prefix Tuning | Prefix Tuning · In-domain SST-2 | accuracy: 87.39 -> 87.29 | No clear effect |
| Add both concentration losses to P-Tuning v2 | P-Tuning v2 · In-domain SST-2 | accuracy: 86.71 -> 87.03 | No clear effect |
| Add both concentration losses to DP₂O | DP₂O · In-domain sentiment average | accuracy: 90.99 -> 90.65 | No clear effect |
| Add both concentration losses to Prompt Tuning | Prompt Tuning · GPT-2 decoder-only backbone; multi-source domain generalization | accuracy: 56.29 -> 57.85; 67.57 -> 68.52; 58.50 -> 59.73; 42.64 -> 42.79; 49.71 -> 50.27; 49.48 -> 51.10 | Helped |
Surprising findings
- Using only the strength loss or only the fluctuation loss sometimes harms soft prompt performance, e.g. on QNLI as target, so both terms are needed together.
- Hard prompts generally outperform soft prompts in the multi-source DG setting (best hard average 73.88% vs. 64.61% for soft).
Limitations
- Pilot experiments cover only a limited variety of prompts.
- Evaluation is limited to few-shot setups (32 samples per source domain) and classification tasks; not open-ended generation.
- Discrete method is input-level only and not extended to distribution-level prompt optimization.
Experimental setup
Datasets: SST-2 MR CR QNLI WNLI RTE ROC SCT COPA
Benchmarks: SST-2 MR CR QNLI WNLI RTE ROC SCT COPA
Metrics: accuracy standard deviation across seeds in-domain/out-of-domain accuracy gap
Baselines: Prompt Tuning Prefix Tuning P-Tuning v2 GrIPS RLPrompt Manual Prompt In-Context Demo DP2O
Builds on: prompt tuning prefix tuning P-Tuning v2 DP2O MAPPO
Closest papers by meaning
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.89)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.89)
- Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning (AAAI 2024 · similarity 0.88)
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.88)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.