Research map / Label-efficient and robust learning / Domain adaptation and generalization
BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BayesPrompt, which fits a Gaussian mixture to PLM encoder representations, refines particles with SVGD toward a debiased domain distribution, and uniformly samples them to initialize prompt tokens for relation extraction with RoBERTa-large.
Problem
Prompt-tuning of pre-trained language models fails to generalize in few-shot settings because the learned prompts are trained on biased, limited samples, so PLMs mis-locate target-domain knowledge; the paper targets few-shot relation extraction.
Approach
Approximate the debiased factual distribution of the downstream domain by fitting a GMM to sample representations and transporting particles with Stein Variational Gradient Descent, then uniformly sample representative features from the result to initialize type prompt tokens alongside label-word prompts.
Key results
- In the 1-shot setting on SemEval, BayesPrompt reaches 35.1 F1 vs 28.6 for KnowPrompt and 33.3 for RetrievalRE.
- Standard setting: BayesPrompt averages 84.5 F1 across four RE benchmarks, versus 84.3 for RetrievalRE and 84.1 for KnowPrompt.
- Paper reports average gains of 3.24% over KnowPrompt across benchmarks in few-shot settings; removing type prompt words drops 1-shot TACRED from 22.5 to 20.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · SemEval K=1 | F1 (%): 33.3(±1.6) -> 35.1(±2.9) | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · SemEval K=5 | F1 (%): 69.7(±1.7) -> 71.6(±3.3) | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · SemEval K=16 | F1 (%): 81.8(±1.0) -> 81.8(±1.2) | No clear effect |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · TACRED few-shot settings | F1 (%): 19.5(±1.5) -> 22.5(±2.5); 30.7(±1.7) -> 31.4(±0.6); 36.1(±1.2) -> 36.2(±0.8) | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · TACREV few-shot settings | F1 (%): 18.7(±1.8) -> 21.9(±2.0); 30.6(±0.2) -> 31.2(±0.8); 35.3(±0.3) -> 35.6(±0.7) | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · SemEval standard | F1 (%): 90.4 -> 90.6 | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · TACRED standard | F1 (%): 72.7 -> 72.9 | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · TACREV standard | F1 (%): 82.7 -> 83.0 | Helped |
| apply BayesPrompt instead of RetrievalRE | RoBERTa-large relation extraction · ReTACRED standard | F1 (%): 91.5 -> 91.4 | Hurt |
| add type prompt words (ablation) | BayesPrompt · TACRED K=1 | F1 (%): 20.2 -> 22.5 | Helped |
| use 19 rather than 9 GMM components (ablation) | BayesPrompt · SemEval K=1 | F1 (%): 33.7(±3.6) -> 35.1(±2.9) | Helped |
| use 19 rather than 9 GMM components (ablation) | BayesPrompt · SemEval K=5 | F1 (%): 70.3(±2.8) -> 71.6(±3.3) | Helped |
Surprising findings
- A Gaussian distribution as the known distribution underperforms a Gaussian mixture model, and the GMM advantage shrinks as shot count grows.
Limitations
- Training complexity is higher than baselines (SVGD and per-step sampling overhead).
- Distribution approximation may be further optimized; method is evaluated only on relation extraction.
Experimental setup
Datasets: SemEval 2010 Task 8 TACRED TACREV RE-TACRED
Benchmarks: SemEval 2010 Task 8 TACRED TACREV RE-TACRED
Metrics: F1 standard deviation t-test p-value
Baselines: FINE-TUNING GDPNET PTR KnowPrompt RetrievalRE SpanBERT KnowBERT LUKE MTB
Builds on: KnowPrompt PTR Stein Variational Gradient Descent
Closest papers by meaning
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.88)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.88)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.88)
- Large Language Model Unlearning via Embedding-Corrupted Prompts (NeurIPS 2024 · similarity 0.88)
- Understanding Prompt Tuning and In-Context Learning via Meta-Learning (NeurIPS 2025 · similarity 0.88)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.