Research map / Large language models / Language, safety and interpretability
Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP2O, which generates a readable discrete prompt set via multi-round GPT-4 dialogue, screens prompts with a supervised+unsupervised entropy metric (SUE), and trains a small policy-gradient network to match prompts to inputs for few-shot classification with RoBERTa-large.
Problem
Discrete prompt optimization for few-shot NLP currently relies on manually designed base prompts and label-supervised screening, while continuous prompt methods are computationally expensive and unreadable.
Approach
Generate candidate prompts by multi-round GPT-4 dialogue aligned to seed inputs, screen them with a linear-complexity SUE metric combining prediction accuracy margin and prediction entropy, and use an RL policy network (policy gradient) to select a prompt per input, ensembling top-k prompt predictions weighted by policy probabilities.
Key results
- DP2O averages 91.80% accuracy across SST-2, MR, CR, and Yelp versus 90.28% for RLPrompt (SOTA baseline), a 1.52-point gain.
- DP2O uses a 0.62M-parameter policy network and cuts training time on SST-2 to 23.75 min versus 218.63 min for RLPrompt (about 10.86%).
- On GLUE few-shot tasks, DP2O improves over Soft Prompt Tuning, Black-Box Tuning, Manual Prompt and In-Context Demo, with gains of 2.7% on QNLI and 5.4% on MRPC over the best baseline reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RLPrompt with DP2O | RoBERTa-large few-shot text classification · SST-2 | accuracy: 90.87 ± 0.86 -> 93.62 ± 0.72 | Helped |
| replace RLPrompt with DP2O | RoBERTa-large few-shot text classification · MR | accuracy: 86.85 ± 0.51 -> 88.58 ± 0.91 | Helped |
| replace RLPrompt with DP2O | RoBERTa-large few-shot text classification · CR | accuracy: 89.62 ± 1.36 -> 90.76 ± 0.50 | Helped |
| replace RLPrompt with DP2O | RoBERTa-large few-shot text classification · Yelp | accuracy: 93.78 ± 2.98 -> 94.25 ± 0.41 | Helped |
| add multi-round dialogue alignment for prompt generation (ablation) | DP2O prompt generation · SST-2 | accuracy: +3.12% | Helped |
| add multi-round dialogue alignment for prompt generation (ablation) | DP2O prompt generation · MR | accuracy: +2.46% | Helped |
| add multi-round dialogue alignment for prompt generation (ablation) | DP2O prompt generation · CR | accuracy: +0.56% | Helped |
| add multi-round dialogue alignment for prompt generation (ablation) | DP2O prompt generation · Yelp | accuracy: +1.07% | Helped |
| combine supervised and unsupervised information for prompt screening (ablation) | DP2O prompt selection · SST-2 | accuracy: +0.59% | Helped |
| use unsupervised information only for prompt screening (ablation) | DP2O prompt selection · four sentiment datasets | accuracy | No clear effect |
| match prompts to inputs with reinforcement learning instead of random matching (ablation) | DP2O prompt matching · SST-2 | accuracy: +1.90% | Helped |
| match prompts to inputs with reinforcement learning instead of similarity-based matching (ablation) | DP2O prompt matching · four sentiment datasets | accuracy | Helped |
Surprising findings
- Prompts selected only with the unsupervised component of SUE perform comparably to supervised selection, suggesting potential for zero-shot use.
- Pseudo-labeled GPT-4 examples are used without verifying label correctness, since prior work reports label authenticity has little impact on PLM performance.
Limitations
- Relies on GPT-4 API for prompt generation, adding cost and dependence on a closed model.
- Experiments limited to classification tasks with a single base PLM family (RoBERTa) and 16-shot-per-class settings.
Experimental setup
Datasets: SST-2 Yelp MR CR RTE QNLI MRPC
Benchmarks: SST-2 Yelp MR CR RTE QNLI MRPC
Metrics: accuracy standard deviation training time
Baselines: Soft Prompt Tuning Black-Box Tuning AutoPrompt Manual Prompt Instruction In-Context Demo GrIPS RLPrompt
Builds on: RLPrompt Black-Box Tuning GrIPS AutoPrompt
Closest papers by meaning
- Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models (NeurIPS 2024 · similarity 0.88)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.88)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.88)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.88)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.88)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.