Research map / Large language models / Language, safety and interpretability
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate label-conditioned training text and fine-tune a bidirectional classifier rather than classify directly by prompting | zero-shot NLU · GLUE classification tasks | average: 50.1 -> 69.4 | Helped |
| rank generated examples by their average log generation probability rather than selecting randomly (ablation) | SuperGen · MNLI-m | accuracy: 63.7 -> 72.3 | Helped |
| apply label smoothing instead of one-hot labels (ablation) | SuperGen · MNLI-m | accuracy: 70.7 -> 72.3 | Helped |
| use temporal ensembling in the training objective (ablation) | SuperGen · MNLI-m | accuracy: 62.0 -> 72.3 | Helped |
| rank generated examples by generation probability rather than selecting randomly (ablation) | SuperGen · GLUE classification tasks | average: 65.1 -> 69.4 | Helped |
| apply label smoothing instead of one-hot labels (ablation) | SuperGen · GLUE classification tasks | average: 65.2 -> 69.4 | Helped |
| use temporal ensembling in the training objective (ablation) | SuperGen · GLUE classification tasks | average: 65.3 -> 69.4 | Helped |
| add label smoothing (ablation) | SuperGen · MRPC | F1: 83.0 -> 82.2 | Hurt |
| use more generated training examples | SuperGen · MNLI-m and SST-2 | accuracy: 100; 10,000 | Mixed |
| continue training on generated data after few-shot training | SuperGen in few-shot settings · MNLI | accuracy: 32-64; 128; 256 | Mixed |
| replace CTRL with GPT-2 as the generator | SuperGen · MNLI-m | accuracy: 72.3 -> 69.5 | Hurt |
Closest papers by meaning
- Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions (ICLR 2024 · similarity 0.89)
- Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning (CVPR 2024 · similarity 0.88)
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.88)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.88)
- Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning (AAAI 2024 · similarity 0.88)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.