Research map / Large language models / Language, safety and interpretability
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use permutation language modeling with XLNet rather than BERT-Base pretraining | XLNet-Base versus BERT-Base · RACE | accuracy: 64.3 -> 66.66 | Helped |
| Add Transformer-XL rather than using BERT-Base architecture with DAE | DAE pretraining · RACE | accuracy: 64.3 -> 65.03 | Helped |
| Use XLNet instead of RoBERTa | RACE test · RACE | accuracy: 83.2 -> 85.4 | Helped |
| Use XLNet instead of RoBERTa | SQuAD2.0 dev · SQuAD2.0 dev | F1: 89.4 -> 90.6 | Helped |
| Use XLNet instead of RoBERTa | SQuAD1.1 dev · SQuAD1.1 dev | F1: 94.6 -> 95.1 | Helped |
| Set partial prediction K to 7 rather than 6 (ablation) | XLNet-Base · RACE | accuracy: 66.66 -> 66.05 | Hurt |
| Add memory caching (ablation) | XLNet-Base (K=6) · RACE | accuracy: 65.55 -> 66.66 | Helped |
| Add span-based prediction (ablation) | XLNet-Base (K=6) · RACE | accuracy: 65.95 -> 66.66 | Helped |
| Use a bidirectional input pipeline (ablation) | XLNet-Base (K=6) · RACE | accuracy: 66.34 -> 66.66 | Helped |
| Add next-sentence prediction (ablation) | XLNet-Base (K=6) · SQuAD2.0 | F1: 80.98 -> 79.83 | Hurt |
Closest papers by meaning
- MPNet: Masked and Permuted Pre-training for Language Understanding (NeurIPS 2020 · similarity 0.90)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.86)
- When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.85)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.85)
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.85)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.