Research map / Large language models / Language, safety and interpretability
Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LM-GC, which serializes 32-bit gradients into hexadecimal text with separators and uses a frozen pretrained LLM's token probabilities in arithmetic coding for zero-shot lossless gradient compression.
Problem
Lossless compression of neural network gradients for distributed/federated learning, which needs a good probability model of high-dimensional, structured gradient data; existing codecs are designed for other modalities.
Approach
Use off-the-shelf pretrained LLMs (zero-shot, no weight updates) as priors for arithmetic coding over gradients converted into grouped hexadecimal text with separators, which better matches the floating-point structure and tokenization LLMs understand.
Key results
- LM-GC with TinyLlama and hex-with-spaces serialization reaches 71.90% compression rate on TinyImageNet gradients vs 86.88% for the best baseline (FPZIP), a 17.2% improvement.
- On CIFAR-10 ConvNet gradients, LM-GC (H_s) with LLaMA 2 7B achieves 32.21% (H_c+s) versus 41.26% for FPZIP and 42.42% for GZIP.
- Serialization matters: ISO-8859-1 byte encoding with LLaMA 2 7B yields 109.07% (expansion), while hex with separators is far better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LM-GC with Tinyllama 1.1B and hexadecimal serialization with spaces instead of the strongest traditional codec | CIFAR-10 ConvNet gradients · CIFAR-10 ConvNet | compression rate (%): 41.26±0.8 -> 38.83±0.4 | Helped |
| Add hexadecimal formatting and spaces instead of using ISO-8859-1 (ablation) | LM-GC with LLAMA 2 7B · CIFAR-10 ConvNet | compression rate (%): 109.07±0.2 -> 32.26±0.5 | Helped |
| Add spaces as separators to hexadecimal serialization (ablation) | LM-GC with LLAMA 2 7B · CIFAR-10 ConvNet | compression rate (%): 72.10±0.5 -> 32.26±0.5 | Helped |
| Use a larger language model (ablation) | LM-GC · CIFAR-10 ConvNet | compression rate (%): 117.38±0.0; 71.85±0.2; 109.07±0.2 | Helped |
| Use hexadecimal serialization with spaces instead of no separators (ablation) | LM-GC with Tinyllama 1.1B · CIFAR-10 ConvNet | compression rate (%): 36.30±0.8 -> 38.83±0.4 | Mixed |
| Use hexadecimal serialization with spaces instead of no separators (ablation) | LM-GC with Tinyllama 1.1B · CIFAR-10 VGG-16, ResNet-18 and ViT | compression rate (%) | Helped |
| Increase the context window size (ablation) | LM-GC with LLAMA 2 7B · Gradient compression | compression rate (%) | Helped |
| Group bytes using BPG 1, 2 or 4 rather than BPG 3 or no grouping (ablation) | LM-GC with Tinyllama and LLAMA 2 · Gradient compression | compression rate (%) | Helped |
| Use smaller bytes-per-group settings, which add separators (ablation) | LM-GC · Gradient compression | computation overhead | Hurt |
| Add LM-GC after quantization or sparsification | Gradient compression pipeline · Quantization and sparsification settings | compression rate | Helped |
| Apply run-length encoding instead of LM-GC | ConvNet gradients · TinyImageNet | compression rate (%): 450.28±0.3; 278.08±0.2; 198.57±0.0; 71.90±0.0 | Hurt |
Surprising findings
- Naive serialization (ISO-8859-1 or hex without separators) can expand data beyond the original size, with LLaMA 2 7B and Tinyllama giving rates above 100%.
- Run-length encoding fails to compress gradients and increases data size.
Limitations
- Throughput is slow: about 4 hours to compress 28 MB, bottlenecked by LLM inference and CPU arithmetic coding.
- Evaluated only on small image-classification settings (ConvNet, VGG-16, ResNet-18, ViT on MNIST/CIFAR-10/TinyImageNet) with a limited number of sub-sampled checkpoints (10).
Experimental setup
Datasets: MNIST CIFAR-10 TinyImageNet
Benchmarks: CIFAR-10 gradients MNIST gradients TinyImageNet gradients
Metrics: compression rate (%) token count
Baselines: PNG FLAC GZIP LZMA FPZIP CABAC run-length encoding SignSGD
Builds on: arithmetic coding Torchac
Closest papers by meaning
- Language Modeling Is Compression (ICLR 2024 · similarity 0.92)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.90)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
- The Fine-Grained Complexity of Gradient Computation for Training Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.