Research map / Large language models / Language, safety and interpretability
Radio: Rate–Distortion Optimization for Large Language Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames LLM weight quantization as a rate-distortion optimization problem and solves it post-training with a stochastic dual-ascent method that assigns per-group bit depths (0–8 bits) plus companded scalar quantization.
Problem
Post-training compression of LLM weights to a user-specified average bit rate while minimizing output distortion; existing mixed-precision methods use few bit-depth options or only per-layer assignment, and OBS/GPTQ-style methods require weight fine-tuning.
Approach
Derives first-order rate-distortion optimality conditions for bit depth allocation (distortion derivative equal across groups, dual variable for average-rate constraint), estimates gradient variances stochastically via PCA-projected backpropagation, and uses a Laplace-motivated companding sigmoid with integer rounding; no weight fine-tuning is needed.
Key results
- At 3 bits, Radio reduces WikiText2 perplexity of OPT-125M to 30.71 vs. 53.43 for GPTQ and 34.43 for QuIP, with minor gains (0.00-0.01) over the next best method on OPT-66B and Llama-2-70B.
- At 2.1-2.8 bits on Llama-2-7B, Radio/256 gets WikiText2 perplexity 9.47-6.21 versus OWQ's 39.56-10.24.
- Radio quantizes Llama-2-7B in 47 minutes on an A100 (vs. 18 minutes for OWQ/GPTQ); a custom kernel gives 1.4-3.3x matvec speedup and 3.8x on OPT-175B at 3 bits on A6000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Radio instead of OWQ for 3-bit quantization | Llama 2 7B · WikiText2 test | perplexity: 6.21 -> 6.04 | Helped |
| Use Radio/256 instead of AWQ/256 | Llama-2 7B · GSM8K | accuracy: 6.97 -> 7.81 | Helped |
| Use Radio/256 instead of AWQ/256 | Llama-2 7B · Average QA | accuracy: 62.48 -> 62.82 | Helped |
| Use Radio/256 instead of AWQ/256 | Llama-2 7B · ARC Challenge | accuracy: 41.13 -> 41.21 | Helped |
| Use Radio/256 instead of GPTQ/256 | Llama-2 7B · HellaSwag | accuracy: 53.02 -> 53.95 | Helped |
| Vary minibatch size across the reported range (ablation) | Radio on OPT-1.3B and OPT-13B · C4 test | perplexity: 16.24 -> 16.22; 12.12 -> 12.11 | No clear effect |
| Vary tokens per sequence across the reported range (ablation) | Radio on OPT-1.3B and OPT-13B · C4 test | perplexity: 16.40 -> 16.21; 12.29 -> 12.10 | No clear effect |
| Reduce group size from 512 to 64 (ablation) | Radio on OPT-1.3B · C4 test, 4 bits | perplexity: 16.22 -> 16.16 | Helped |
| Reduce group size from 512 to 64 (ablation) | Radio on OPT-1.3B · C4 test, 3 bits | perplexity: 16.86 -> 16.62 | Helped |
| Add MMSE step sizes to RTN (ablation) | OPT-1.3B quantization · C4, 4 bits | perplexity: 24.51 -> 16.98 | Helped |
| Add mixed-precision depths (ablation) | OPT-1.3B quantization with MMSE step sizes · C4, 4 bits | perplexity: 16.98 -> 16.29 | Helped |
| Add companding (ablation) | OPT-1.3B quantization with mixed-precision depths · C4, 4 bits | perplexity: 16.29 -> 16.22 | Helped |
Surprising findings
- RTN at 3 bits on Llama-2 models collapses on GSM8K (1.82-6.14%) despite comparable WikiText2 perplexity to Radio, showing perplexity alone can hide downstream degradation.
- Increasing calibration examples from 128 to 1024 did not noticeably change C4 perplexity (±0.01).
Limitations
- Quantization runtime grows near linearly with model size; 70B takes about 11 hours, which is slower than several baselines.
- Gains over strong baselines are minor (0.00-0.01 PPL) for the largest models at 3 bits.
- Results reported mainly with perplexity; downstream evaluation is limited to Llama-2 at 3 bits.
Experimental setup
Datasets: C4
Benchmarks: WikiText2 C4 GSM8K ARC HellaSwag PIQA WinoGrande
Metrics: perplexity accuracy running time
Baselines: RTN GPTQ QuIP OWQ AWQ OmniQuant SqueezeLLM
Builds on: GPTQ Optimal Brain Surgeon AWQ OWQ
Closest papers by meaning
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.90)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.89)
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment (NeurIPS 2024 · similarity 0.89)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.89)
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.89)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.