Research map / Large language models / Language, safety and interpretability
Language Modeling Is Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that large pretrained language models used as arithmetic-coding compressors achieve strong in-context compression across text, image, and audio, and analyzes scaling, tokenization, and generative sampling through the compression lens.
Problem
Whether the prediction-compression equivalence can make foundation models into general-purpose lossless compressors, and how compression perspective informs scaling laws, tokenization, and in-context learning.
Approach
Treat large language models as offline (in-context) arithmetic-coding compressors, evaluating raw and parameter-adjusted compression rates against classical compressors across modalities; also use compressors (gzip) as generative models via the reverse equivalence.
Key results
- Chinchilla 70B compresses ImageNet patches to 48.0% raw (43.4% in intro) and LibriSpeech to 21.0% raw (16.4% in abstract), beating PNG (58.5%) and FLAC (30.3%) on those modalities.
- Adjusted compression rates for billion-parameter models exceed 1000% on 1GB datasets because model size (2 bytes/parameter) dominates, so large models only pay off on datasets in the TB range.
- Among enwik-trained Transformers, the optimal model size grows with test-set size; on small datasets larger models compress worse under adjusted rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chinchilla 70B instead of LZMA2 for arithmetic-coded compression | Chinchilla 70B · enwik9 | raw compression rate (%): 23.0 -> 8.3 | Helped |
| Use Chinchilla 70B instead of LZMA2 for arithmetic-coded compression | Chinchilla 70B · ImageNet | raw compression rate (%): 57.9 -> 48.0 | Helped |
| Use Chinchilla 70B instead of FLAC for arithmetic-coded compression | Chinchilla 70B · LibriSpeech | raw compression rate (%): 30.3 -> 21.0 | Helped |
| Include model parameter size in the compressed output | Chinchilla models · 1GB datasets | adjusted compression rate (%): null | Hurt |
| Scale model size upward (ablation) | Transformers trained on enwik8 · larger datasets | adjusted compression rate (%) | Helped |
| Scale model size upward (ablation) | Transformers trained on enwik8 · smaller enwik datasets | adjusted compression rate (%) | Hurt |
| Increase the BPE vocabulary size (ablation) | Small Transformers trained on enwik8 · enwik8 | compression rate | Helped |
| Increase the BPE vocabulary size (ablation) | Bigger Transformers trained on enwik8 · enwik8 | compression rate | Hurt |
Surprising findings
- Chinchilla 70B, trained mainly on text, outperforms PNG and FLAC on image and audio data in raw compression rate.
- Adjusted compression rates for large LLMs are far above 100% on 1GB data because parameter size dominates the code length.
Limitations
- Adjusted compression of large models is poor on 1GB datasets because of parameter overhead.
- Offline two-part coding only; prequential/online coding not evaluated for LLMs.
- Chinchilla is not open-weight and training data is not fully known, so generalization to unseen modalities is hard to verify.
Experimental setup
Datasets: enwik8 enwik9 ImageNet LibriSpeech
Benchmarks: enwik9 ImageNet LibriSpeech Random data
Metrics: raw compression rate adjusted compression rate
Baselines: gzip LZMA2 PNG FLAC
Builds on: arithmetic coding Solomonoff induction Chinchilla scaling laws
Closest papers by meaning
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.92)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Accuracy is Not All You Need (NeurIPS 2024 · similarity 0.90)
- EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.