AI Paper Atlas

Research map / Benchmarks / Penn Treebank

Penn Treebank leaderboard

50 papers at ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR report results on Penn Treebank. Best perplexity: 19.45 (GPT2 + GFSA, NeurIPS 2024).

Numbers are as each paper reports them, read by AI from its results tables and not checked by hand. Papers can differ in setting (training data, model size, extra data); the topic column shows what each paper is about. Open the paper before you compare or quote a number.

Penn Treebank · perplexity

37 papers, each with its best reported number for its own method; lower is better.

#perplexity ↓MethodPaperTopic
119.45GPT2 + GFSA
GPT2
Graph Convolutions Enrich the Self-Attention in Transformers!
NeurIPS 2024 · Table: Results comparison on G
Attention and transformers
244.3LasFModeling Language Tokens as Functionals of Semantic Fields
ICML 2024 · Table 4
Attention and transformers
347.6AWD-LSTM-MoS + Noisin + dynamic evaluation
AWD-LSTM-MoS + dynamic evaluation
Noisin: Unbiased Regularization for Recurrent Neural Networks
ICML 2018 · Table 3
RNNs and normalization
447.69AWD-LSTM-MoS + dynamic evaluation
AWD-LSTM
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
ICLR 2018 · Table 1
Language, safety and interpretability
551.1AWD-LSTM + dynamic eval
AWD-LSTM (ours)
Dynamic Evaluation of Neural Sequence Models
ICML 2018 · Penn Treebank perplexities tab
RNNs and normalization
652.8AWD-LSTM + continuous cache pointer
AWD-LSTM (tied)
Regularizing and Optimizing LSTM Language Models
ICLR 2018 · Table 1
RNNs and normalization
754.2AWD-LSTM + DeFINE
AWD-LSTM
DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence Modeling
ICLR 2020 · Table: Comparison with existin
Attention and transformers
854.3LSTM + MbPA
LSTM
Memory-based Parameter Adaptation
ICLR 2018 · Table: PTB and WikiText-2 perp
Continual fine-tuning of LLMs
955.3aLSTM, recurrent policy, 24MBreaking the Activation Function Bottleneck through Adaptive Parameterization
NeurIPS 2018 · Table 2
RNNs and normalization
1055.7AWD-LSTM-DRILL
AWD-LSTM
Deep Residual Output Layers for Neural Language Generation
ICML 2019 · Penn Treebank results table
Attention and transformers
27 more papers on this leaderboard, the baselines reported on it and a CSV download are free with a scime account.
Sign in free to see all

Penn Treebank · perplexity (val)

14 papers, each with its best reported number for its own method; lower is better.

#perplexity ↓MethodPaperTopic
158.29ON-LSTM 3-layer (tied)Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
ICLR 2019 · Table 1
RNNs and normalization
258.5ShrinkRNNAutoShrink: A Topology-Aware NAS for Discovering Efficient Neural Architecture
AAAI 2020 · Table: Performance comparison
Neural architecture search
359.2MC-RNN
LSTM
Modeling Local Dependence in Natural Language with Multi-Channel Recurrent Neural Networks
AAAI 2019 · Table 3
RNNs and normalization
459.4Fraternal dropout
AWD-LSTM 3-layer
Fraternal Dropout
ICLR 2018 · FD vs ELD comparison table
RNNs and normalization
572.4Pointer Sentinel-LSTM (medium)
LSTM
Pointer Sentinel Mixture Models
ICLR 2017 · Table: Comparison over Penn Tr
Attention and transformers
672.92-layer RevLSTM (no limit)Reversible Recurrent Neural Networks
NeurIPS 2018 · Table: Validation perplexities
RNNs and normalization
774.6Linear interpolation neural cache
LSTM
Improving Neural Language Models with a Continuous Cache
ICLR 2017 · text
RNNs and normalization
879.9CharLM with dynamic skip, λ=1, K=5
CharLM
Long Short-Term Memory with Dynamic Skip Connections
AAAI 2019 · Table 2
RNNs and normalization
980.24ISSLearning Intrinsic Sparse Structures within Long Short-Term Memory
ICLR 2018 · Table 1
RNNs and normalization
1082.9QRNN (medium)Quasi-Recurrent Neural Networks
ICLR 2017 · Table
RNNs and normalization
4 more papers on this leaderboard, the baselines reported on it and a CSV download are free with a scime account.
Sign in free to see all

Penn Treebank · test perplexity

10 papers, each with its best reported number for its own method; lower is better.

#test perplexity ↓MethodPaperTopic
154.89Few-shot DARTS
DARTS
Few-Shot Neural Architecture Search
ICML 2021 · text
Neural architecture search
257.25LMS-PLIF, 10^5 knots
AWD-LSTM
Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities
ICML 2019 · Table 1
RNNs and normalization
357.59R-DARTS(L2)
DARTS
Understanding and Robustifying Differentiable Architecture Search
ICLR 2020 · Table: DARTS vs. RobustDARTS o
Neural architecture search
458.8Word-level LSTM BA
Word-level LSTM
Augment Your Batch: Improving Generalization Through Instance Repetition
CVPR 2020 · Table 2
Optimizers
558.9661Our Algorithm
3-layer AWD-LSTM
Robustness to Unbounded Smoothness of Generalized SignSGD
NeurIPS 2022 · Table 1
Optimizers
664.4AdamW-Win2
3-layer LSTM
Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training
JMLR 2024 · Table 5
Optimizers
764.8PowerSign
10M single-layer LSTM
Neural Optimizer Search with Reinforcement Learning
ICML 2017 · Table: Language modeling
Optimizers
873.3IMM with KN Bigram
LSTM
Induced Model Matching: Restricted Models Help Train Full-Featured Models
NeurIPS 2024 · Table 1
Knowledge distillation
981.67SGD with Momentum, batch size 64
word-level LSTM
On the Generalization Benefit of Noise in Stochastic Gradient Descent
ICML 2020 · Appendix table, constant step
Implicit bias and generalization
1083.86HFP8 + Round-off update
2-LSTM
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks
NeurIPS 2019 · Table 4
Quantization

Not affiliated with any conference, journal or the benchmark's authors. Corrections: contact@latenticlabs.com.