Research map / Architectures and efficiency / RNNs and normalization
L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L3TC, a low-complexity learned lossless text compressor built on a small RWKV model with an outlier-aware tokenizer and high-rank reparameterization (HiRA), reaching 48% bit saving over gzip on enwik9 with MB/s decoding.
Problem
Learned probabilistic compressors based on large language models achieve strong ratios but have prohibitive decoding complexity and model size, making them impractical for real-time text compression.
Approach
Use a small RWKV backbone for fast per-token prediction, pair it with a tokenizer that covers frequent tokens with a limited vocabulary while passing rare characters (outliers) through a raw UTF-8 bypass, and train with extra low-rank-style parallel branches (HiRA) that are merged into the weights for inference.
Key results
- L3TC-3.2M achieves 16.87% ACR on enwik9, a 48% bit saving relative to gzip (32.26%), with a 50x reduction in parameters versus other learned compressors.
- L3TC-200K decodes at 1.30 MB/s on iPhone12 ANE and 4.35 MB/s on an A100, the fastest among the learned compressors compared.
- HiRA reduces CR of RWKV-200K from 24.36% to 23.51% with no added MACs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L3TC-3.2M instead of gzip | enwiki9 compression · enwiki9 | ACR (%): 32.26 -> 16.87 | Helped |
| Increase L3TC model size from 3.2M to 12M (ablation) | L3TC · enwiki9 | CR (%): 16.23 -> 16.00 | No clear effect |
| Use L3TC-12M instead of gzip | enwiki8 compression · enwiki8 | CR (%): 36.45 -> 11.24 | Helped |
| Use RWKV instead of Transformer or Transformer-XL | 200K-parameter language model · enwiki9 | Time per token: 8.61ms -> 0.34ms; 4.36ms -> 0.34ms | Helped |
| Set tokenizer coverage to 0.999 (ablation) | outlier-aware tokenizer · enwik8 tokenizer evaluation | bits per byte | Helped |
| Increase vocabulary size beyond 16K (ablation) | outlier-aware tokenizer · enwiki9 | CR (%) | Hurt |
| Increase HiRA rank (ablation) | RWKV with HiRA · enwiki9 | CR (%) | Helped |
| Increase the number of HiRA branches (ablation) | RWKV with HiRA · enwiki9 | CR (%) | Helped |
| Add a linear shortcut (ablation) | RWKV-200K · enwiki9 | CR (%): 24.36 -> 24.18 (-0.18) | Helped |
| Add HiRA to RWKV with a linear shortcut (ablation) | RWKV-200K + Linear · enwiki9 | CR (%): 24.18 -> 23.51 | Helped |
| Add HiRA while retaining the character-based tokenizer (ablation) | RWKV-200K · iPhone12 ANE | Decoding speed: 376 KB/s -> 376 KB/s | No clear effect |
| Replace the character-based tokenizer with the outlier-aware tokenizer (ablation) | L3TC-200K · iPhone12 ANE | Decoding speed: 376 to 842 KB/s; 2.2× | Helped |
Surprising findings
- Pretrained Llama2-13B achieves the lowest raw CR on enwik9, but its model size makes ACR around 2608%, so ACR reverses the ranking.
- L3TC-12M shows no further gain over L3TC-3.2M on enwik9, which the authors attribute to overfitting when model size approaches training data size.
Limitations
- Larger models (e.g., L3TC-12M) overfit and do not improve on enwik9 relative to the 3.2M model.
- Compression ratio lags behind large pretrained LLM compressors and cmix/nncp in raw CR.
Experimental setup
Datasets: enwik8 enwik9
Benchmarks: enwik9 enwik8
Metrics: compression ratio (CR) adjusted compression ratio (ACR) MACs decoding speed bits per byte
Baselines: gzip bzip2 zstd tszip Llama2-7B Llama2-13B Chinchilla RWKV-1.5B RWKV-7B Transformer Transformer-XL CMIX v20
Builds on: RWKV LoRA byte pair encoding NNCP LLMZip
Closest papers by meaning
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.86)
- Extreme Compression of Large Language Models via Additive Quantization (ICML 2024 · similarity 0.86)
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction (NeurIPS 2026 · similarity 0.85)
- Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.85)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.85)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.