Research map / Large language models / Language, safety and interpretability
BERTs are Generative In-Context Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that masked language models (DeBERTa) can perform generative in-context learning via a simple input-reformatting inference trick (appending [MASK] tokens) and an interpolated pseudo-log-likelihood ranking, without any training, and compares them to GPT-3 across SuperGLUE, translation, QA and commonsense tasks.
Problem
Masked language models are assumed to be unable to perform generative in-context learning, which is usually associated only with causal LMs such as GPT-3.
Approach
Repurpose an off-the-shelf DeBERTa for left-to-right generation by appending [MASK] tokens (plus two extra masks before [SEP]) and for ranking by a PLL variant that masks two extra right-context tokens; no additional training or architectural change is needed.
Key results
- DeBERTa 1.5B 1-shot SuperGLUE average 68.4 vs GPT-3 175B 1-shot 68.9 (GPT-3 1.3B-size comparison 57.8 average in table), with DeBERTa clearly stronger on language understanding.
- On language modeling/Winograd-style tasks, DeBERTa 1.5B few-shot average 75.4 vs GPT-3 64.8.
- On machine translation, DeBERTa is far behind (1-shot average 13.8 vs GPT-3 18.8 SacreBLEU) and on closed-book QA it is weaker (1-shot NQs 2.6 vs 5.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeBERTa with the paper's masked-token inference method rather than GPT-3 | SuperGLUE · SuperGLUE (1-shot average) | accuracy: 57.8 -> 68.4 | Helped |
| Use DeBERTa with the paper's masked-token inference method rather than GPT-3 | Text completion and Winograd-style tasks · Text completion and Winograd-style tasks (1-shot average) | accuracy: 65.9 -> 72.8 | Helped |
| Use DeBERTa with the paper's masked-token inference method rather than GPT-3 | Machine translation · Machine translation (1-shot average) | SacreBLEU: 18.8 -> 13.8 | Hurt |
| Use DeBERTa with the paper's masked-token inference method rather than GPT-3 | Closed-book QA and commonsense reasoning · Closed-book QA and commonsense reasoning (1-shot average) | accuracy: 36.3 -> 33.3 | Hurt |
| Increase autoregressive generation masks from one to three (ablation) | DeBERTa 1.5B · German-to-English translation (1-shot) | SacreBLEU: 10.0 -> 23.7 | Helped |
| Increase autoregressive generation masks from two to three (ablation) | DeBERTa 1.5B · German-to-English translation (1-shot) | SacreBLEU: 22.4 -> 23.7 | Helped |
| Increase autoregressive generation masks from three to four (ablation) | DeBERTa 1.5B · German-to-English translation (1-shot) | SacreBLEU: 23.7 -> 23.9 | Helped |
| Use autoregressive generation instead of Markov-chain Monte-Carlo sampling (ablation) | DeBERTa 1.5B · German-to-English translation (1-shot) | SacreBLEU: 1.6 -> 23.7; 2.6 -> 23.7 | Helped |
| Mask the target and two following subwords instead of only the target subword (ablation) | DeBERTa 1.5B ranking · ReCoRD (zero-shot) | exact match: 80.9 -> 87.1 | Helped |
| Use three masks rather than two for pseudo-log-likelihood ranking (ablation) | DeBERTa 1.5B ranking · ReCoRD (zero-shot) | exact match: 86.0 -> 87.1 | Helped |
| Use four masks instead of three for pseudo-log-likelihood ranking (ablation) | DeBERTa 1.5B ranking · ReCoRD (zero-shot) | exact match: 87.1 -> 86.9 | Hurt |
| Use exact unidirectional log-likelihood instead of three-mask pseudo-log-likelihood (ablation) | DeBERTa 1.5B ranking · ReCoRD (zero-shot) | exact match: 87.1 -> 77.2 | Hurt |
Surprising findings
- DeBERTa outperforms the similarly sized GPT-3 on language understanding and language modeling tasks while lagging on closed-book QA and translation.
- Interpolated PLL with two extra masked right-context tokens improves ReCoRD accuracy over the original PLL (87.1 vs 80.9 EM).
Limitations
- Generation is slower because self-attention keys/values must be recomputed each step.
- Weaker closed-book QA and translation, attributed to a small monolingual training corpus and MLM objective.
- Relies on GPT-3 prompt templates, which may be unfair to DeBERTa; only a single DeBERTa scale is compared in main results.
Experimental setup
Datasets: SuperGLUE WMT14 French-English WMT16 Romanian-English WMT16 German-English Natural Questions Web Questions TriviaQA PIQA ARC OpenBookQA HellaSwag StoryCloze
Benchmarks: SuperGLUE HellaSwag StoryCloze Winograd Winogrande WMT translation pairs Natural Questions Web Questions TriviaQA PIQA ARC-Easy ARC-Challenge
Metrics: accuracy exact match F1 SacreBLEU
Baselines: GPT-3 1.3B GPT-3 175B OPT 1.3B Gibbs sampling for BERT generation finetuned BERT-large
Builds on: GPT-3 DeBERTa pseudo-log-likelihood scoring RULER
Closest papers by meaning
- Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models (ICLR 2025 · similarity 0.87)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.87)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.87)
- Toward Understanding In-context vs. In-weight Learning (ICLR 2025 · similarity 0.86)
- Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting (ICLR 2025 · similarity 0.86)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.