Research map / Large language models / Language, safety and interpretability
Prompting GPT-3 To Be Reliable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use source-domain demos rather than supervised fine-tuning | GPT-3 · MRQA target domains | F1: 62.1 -> 77.2 | Helped |
| Sample demos from target domains rather than source domains (ablation) | GPT-3 · MRQA target domains | F1: 77.2 -> 77.2 | No clear effect |
| Use few-shot demos from clean training data | GPT-3 · AdvGLUE perturbed test sets | accuracy: 51.7 -> 69.3 | Helped |
| Balance pro-bias and anti-bias demonstrations | GPT-3 · WinoBias Type I | absolute bias gap: 51.0 -> 8.1 | Helped |
| Put pro-bias examples at the end rather than randomly shuffling balanced examples (ablation) | GPT-3 · WinoBias Type I | absolute bias gap: 8.1 -> 13.2 | Hurt |
| Use only ambiguous neutral-answer demonstrations (ablation) | GPT-3 · BBQ | disambiguated accuracy: 76.0 -> 13.2 | Hurt |
| Add a fairness instruction and direct the model to choose unknown when evidence is insufficient | GPT-3 · BBQ ambiguous questions | ambiguous bias score: 24.7 -> 1.9 | Helped |
| Use language-model probability as confidence | GPT-3 · NQ | ECE: 29.4 -> 18.9 | Helped |
| Use 64 rather than 2 prompt demonstrations (ablation) | GPT-3 with LM-Prob confidence · NQ | accuracy: 37.0 -> 42.8 | Helped |
| Use 64 rather than 2 prompt demonstrations (ablation) | GPT-3 with LM-Prob confidence · NQ | ECE: 11.7 -> 13.4 | Mixed |
| Provide counterfactual passage-question-answer triples instead of original question-answer pairs (ablation) | GPT-3 · NQ | update rate: 25.8% -> 85.4% | Helped |
| Provide counterfactual passage-question-answer triples instead of original question-answer pairs (ablation) | GPT-3 · SQuAD | update rate: 29.1% -> 84.8% | Helped |
Closest papers by meaning
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.87)
- Understanding the Effects of Iterative Prompting on Truthfulness (ICML 2024 · similarity 0.87)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.87)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.86)
- SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (NeurIPS 2024 · similarity 0.86)
- Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.