AI Paper Atlas

Research map / Benchmarks / Enwik8

Enwik8 leaderboard

12 papers at ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR report results on Enwik8. Best BPC: 0.898 (LSTM, 46M, depth 4, ICLR 2018).

Numbers are as each paper reports them, read by AI from its results tables and not checked by hand. Papers can differ in setting (training data, model size, extra data); the topic column shows what each paper is about. Open the paper before you compare or quote a number.

Enwik8 · BPC

12 papers, each with its best reported number for its own method; lower is better.

#BPC ↓MethodPaperTopic
10.898LSTM, 46M, depth 4On the State of the Art of Evaluation in Neural Language Models
ICLR 2018 · Table: Enwik8 results
RNNs and normalization
20.97CP-Transformer (14)
Transformer
Efficient Representation Learning via Adaptive Context Pooling
ICML 2022 · Table: BPC and model size
Attention and transformers
30.97Transformer-LS
110M
Long-Short Transformer: Efficient Transformers for Language and Vision
NeurIPS 2021 · Char-LM results table
Attention and transformers
40.9724L Compressive Transformer
TransformerXL
Compressive Transformers for Long-Range Sequence Modelling
ICLR 2020 · Table: State-of-the-art result
Attention and transformers
51Top-KAST (80%, 0%) 55M
Transformer-XL
Top-KAST: Top-K Always Sparse Training
NeurIPS 2020 · Table (Enwik8)
RNNs and normalization
61.03GNN-LM + kNN
12L Transformer-XL
GNN-LM: Language Modeling based on Global Contexts via GNN
ICLR 2022 · Table: Bit per Character on th
GNN architectures
71.11LadderStaircase Attention for Recurrent Processing of Sequences
NeurIPS 2022 · Table 1
Attention and transformers
81.1486SMoE-Dropout (k=N)
Transformer-XL
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
ICLR 2023 · Table: pre-training results
Mixture of experts
91.16Memoria Transformer
GPT-2
Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture
ICML 2024 · Language modeling performance
RNNs and normalization
101.1982 × Large FS-LSTM-4Fast-Slow Recurrent Neural Networks
NeurIPS 2017 · Table: BPC on enwik8
RNNs and normalization
2 more papers on this leaderboard, the baselines reported on it and a CSV download are free with a scime account.
Sign in free to see all

Not affiliated with any conference, journal or the benchmark's authors. Corrections: contact@latenticlabs.com.