AI Paper Atlas

Research map / Benchmarks / WikiText-103

WikiText-103 leaderboard

107 papers at ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR report results on WikiText-103. Best perplexity: 3.92 (RETRO, ICML 2022).

Numbers are as each paper reports them, read by AI from its results tables and not checked by hand. Papers can differ in setting (training data, model size, extra data); the topic column shows what each paper is about. Open the paper before you compare or quote a number.

WikiText-103 · perplexity

84 papers, each with its best reported number for its own method; lower is better.

#perplexity ↓MethodPaperTopic
13.92RETROImproving Language Models by Retrieving from Trillions of Tokens
ICML 2022 · Table: Perplexities on Wikitex
Attention and transformers
24.59ESPACE (20%)
Llama2-13B
ESPACE: Dimensionality Reduction of Activations for Model Compression
NeurIPS 2024 · Table 1
Pruning and efficient tuning
35.95LTE (η=2)
LLaMA-7B
Learn To be Efficient: Build Structured Sparsity in Large Language Models
NeurIPS 2024 · Table: Performance comparison
Language, safety and interpretability
47.16ChinchillaAn empirical analysis of compute-optimal large language model training
NeurIPS 2022 · text
Pruning and efficient tuning
58.566.7B+muP
GPT-3 6.7B
Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
NeurIPS 2021 · Table 7
Pruning and efficient tuning
69.76GPT2-large + kNN-LM
GPT2-large 774M
Why do Nearest Neighbor Language Models Work?
ICML 2023 · Table 5 (appendix)
Retrieval and knowledge
710.6Hybrid H3-2.7BHungry Hungry Hippos: Towards Language Modeling with State Space Models
ICLR 2023 · Table 4
Attention and transformers
810.93+MemDec
GPT2-xl
Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
NeurIPS 2025 · Table: Perplexity comparison o
Retrieval and knowledge
914.29LLaMA MHA (α=0.5)
LLaMA
On the Role of Hidden States of Modern Hopfield Network in Transformer
NeurIPS 2025 · Table: LLaMA perplexity
Attention and transformers
1014.8GNN-LM + kNN
base LM
GNN-LM: Language Modeling based on Global Contexts via GNN
ICLR 2022 · Table: Test perplexity on Wiki
GNN architectures
74 more papers on this leaderboard, the baselines reported on it and a CSV download are free with a scime account.
Sign in free to see all

WikiText-103 · perplexity (val)

19 papers, each with its best reported number for its own method; lower is better.

#perplexity ↓MethodPaperTopic
112.87RETROImproving Language Models by Retrieving from Trillions of Tokens
ICML 2022 · Table: Perplexities on Wikitex
Attention and transformers
219.24Gaudi-GBLR
GPT-2
Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks
ICLR 2024 · Table: Perplexity by weight ty
RNNs and normalization
320Adam with untuned exponential warmup
Transformer LM
On the Adequacy of Untuned Warmup for Adaptive Optimization
AAAI 2021 · Table 2
Optimizers
420.69NMST+GPT-2
GPT-2
A Non-monotonic Self-terminating Language Model
ICLR 2023 · Table 2
Combinatorial and search optimization
521.4Lite TransformerLite Transformer with Long-Short Range Attention
ICLR 2020 · Table: Results on WIKITEXT-103
Attention and transformers
622.4Transformer-XL Base + URPE
Transformer-XL Base
Your Transformer May Not be as Powerful as You Expect
NeurIPS 2022 · Table 1
Attention and transformers
723.5cosFormercosFormer: Rethinking Softmax In Attention
ICLR 2022 · Table: Autoregressive language
Attention and transformers
824.3ENT-Threshold
GPT2-large
Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models
ICLR 2024 · Table: Validation perplexity o
Fast diffusion sampling
930.6NPRF-Transformer with RPEStable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
NeurIPS 2021 · Language model perplexity tabl
Attention and transformers
1030.8Relational Memory CoreRelational recurrent neural networks
NeurIPS 2018 · Table 2
Attention and transformers
9 more papers on this leaderboard, the baselines reported on it and a CSV download are free with a scime account.
Sign in free to see all

WikiText-103 · PPL

18 papers, each with its best reported number for its own method; lower is better.

#PPL ↓MethodPaperTopic
116.97CHELAShort-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences
ICML 2024 · Table 4
Attention and transformers
217.9Tensor-Transformer 1core + PN
Tensor-Transformer 1core
PowerNorm: Rethinking Batch Normalization in Transformers
ICML 2020 · Language modeling results tabl
RNNs and normalization
318.9Tensorized Transformer core-2
Transformer
A Tensorized Transformer for Language Modeling
NeurIPS 2019 · Table 2
Attention and transformers
419.51RB-LRA
GPT-2 Medium
LRA-QViT: Integrating Low-Rank Approximation and Quantization for Robust and Efficient Vision Transformers
ICML 2025 · Table 8
Quantization
519.9AdaHessian
Six-layer tensorized Transformer
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
AAAI 2021 · Table 3
Optimizers
620.4FlashButterfly-GPT-2-Small
GPT-2-Small
Simple Hardware-Efficient Long Convolutions for Sequence Modeling
ICML 2023 · Table: Performance of replacin
Attention and transformers
723.471Memoria Transformer
GPT-2
Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture
ICML 2024 · Language modeling performance
RNNs and normalization
823.82SimCTG
GPT-2-small
A Contrastive Framework for Neural Text Generation
NeurIPS 2022 · Table 1
Language, safety and interpretability
924.03TNLVarious Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention
ICML 2024 · Results on Wikitext-103 table
Attention and transformers
1024.1Ouroboros + Adam
Transformer-XL
Ouroboros: On Accelerating Training of Transformer-Based Language Models
NeurIPS 2019 · Test bpc/PPL table
Attention and transformers
8 more papers on this leaderboard, the baselines reported on it and a CSV download are free with a scime account.
Sign in free to see all

Not affiliated with any conference, journal or the benchmark's authors. Corrections: contact@latenticlabs.com.