Research map / Architectures and efficiency / Pruning and efficient tuning
Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that small decoder-only transformers (millions of parameters) pre-trained offline on 165GB of byte-level text, image, or audio data can beat gzip, LZMA2 and domain-specific compressors (PNG, JPEG-XL, FLAC) on out-of-distribution data when model size is counted, but transfer to unseen modalities is weak.
Problem
Whether small pre-trained transformers, with parameter count included in the compression ratio, can compete with standard compressors across modalities and transfer to unseen modalities, as large LLMs reportedly do.
Approach
Instead of large foundation models, train families of small decoder-only transformers offline on byte streams from text, image, and audio (unimodal and multimodal mixtures), freeze them, and use arithmetic coding with adjusted compression ratios that include float16 weight size, evaluated on 1GB OOD data.
Key results
- Best small pre-trained model achieves compression ratio 0.487 on OOD audio vs 0.538 for FLAC and 0.509 for Bellard's online transformer.
- On OOD image data, the pre-trained model reaches 0.285, versus 0.260 for JPEG-XL (best image compressor), so it does not beat JPEG-XL there.
- Out-of-modality transfer is weak: models trained without a modality fail to beat standard compressors on it, unlike LLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed pre-trained transformer instead of Bellard | OOD audio compression · LibriSpeech OOD audio | compression ratio: 0.509 -> 0.487 | Helped |
| Use the proposed pre-trained transformer instead of JPEG-XL | OOD image compression · CelebA HQ OOD images | compression ratio: 0.260 -> 0.285 | Hurt |
| Use the proposed pre-trained transformer instead of Bellard | OOD text compression · Reddit OOD text | compression ratio: 0.204 -> 0.217 | Hurt |
| Train on multiple modalities rather than one modality (ablation) | pre-trained transformers · individual in-training modalities | compression ratio | Hurt |
| Train on multiple modalities rather than one modality (ablation) | pre-trained transformers · multimodal OOD data with all evaluation modalities seen in training | compression ratio | Helped |
| Train on a smaller pure-text dataset instead of a larger text-plus-other-modality mixture (ablation) | pre-trained transformers · text OOD | compression ratio | Helped |
| Increase model and training dataset size (ablation) | unimodal pre-trained transformers · OOD compression | compression ratio | Helped |
| Increase context size (ablation) | pre-trained transformers · text OOD | compression ratio | Hurt |
| Increase context size (ablation) | pre-trained transformers · image OOD | compression ratio | Helped |
| Increase sliding-window overlap (ablation) | pre-trained transformers with context length 4096 · OOD compression | compression ratio | No clear effect |
| Increase evaluation dataset size (ablation) | offline pre-trained neural compressors · OOD compression | compression ratio | Helped |
| Train on multimodal data instead of unimodal data (ablation) | largest tested transformers · image OOD | compression ratio | Helped |
Surprising findings
- Multimodal training only slightly hurts in-modality performance while substantially improving multimodal compression, yet unseen-modality transfer is weak.
- Training on a 33GB pure-text dataset outperforms training on 82.5GB text plus 82.5GB images/audio on text evaluation, i.e., little cross-modal transfer.
Limitations
- Compressor size counts only float16 weights without compressing them, making ratios upper bounds.
- Models are orders of magnitude slower than standard compressors; not practical as a compressor.
- Results limited to the chosen model sizes, modalities and OOD datasets; no transfer-learning methods explored.
Experimental setup
Datasets: Wikipedia PG-19 Big Patent Scientific Papers Natural Instructions ImageNet Common Voice Reddit CelebA HQ LibriSpeech
Benchmarks: Reddit 1GB (OOD text) CelebA HQ 1GB (OOD image) LibriSpeech 1GB (OOD audio)
Metrics: adjusted compression ratio (compressed size + compressor siz
Baselines: gzip LZMA2 FLAC PNG JPEG-XL JPEG 2000 Bellard online transformer
Builds on: Language modeling is compression (Delétang et al.) Bellard online transformer (NNCP-style) Llama
Closest papers by meaning
- CASP: Compression of Large Multimodal Models Based on Attention Sparsity (CVPR 2025 · similarity 0.88)
- Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.87)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.87)
- Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need (NeurIPS 2025 · similarity 0.87)
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models (NeurIPS 2026 · similarity 0.87)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.