Research map / Architectures and efficiency / Quantization
RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RWKVQuant, a PTQ framework for RWKV that uses a coarse-to-fine entropy/central-moment proxy to choose per-layer between scalar quantization (GPTQ) and vector quantization, plus codebook optimization for element-wise multiplication, reaching about 3.275 bpw.
Problem
Post-training quantization of RWKV models degrades badly under existing SQ and VQ methods, because RWKV's non-linear operators block parameter fusion for smoothing/rotation methods and its more uniform weights hurt clustering-based VQ.
Approach
A coarse-to-fine proxy (information entropy of sorted weight gaps plus weighted high-order central moments) adaptively selects SQ (GPTQ) for uniform weights without outliers and VQ otherwise, combined with an activation-weighted KMeans codebook for element-wise multiplication layers.
Key results
- RWKVQuant at 3.275 bpw on RWKV6-14B reaches 62.69 zero-shot average and 2.89 LAMBADA PPL, versus 63.65 and 3.02 for floating point.
- On RWKV6-14B quantized to 3.275 bpw, speed rises from 16.02 to 34.32 tokens/sec (2.14x) and memory drops from 26.07 GB to 9.21 GB (2.83x) on an A6000.
- Under 3.25/3.5 bpw settings, prior SQ and VQ baselines such as GPTQ, GPTVQ and VPTQ show larger perplexity increases; for example, on RWKV7-0.1B the best baseline PPL is around 23.75 versus 18.41 for RWKVQuant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RWKVQuant instead of the strongest reported single-quantization baselines | RWKV language models · RWKV7-0.1B through RWKV6-14B | 0-shot 9-task accuracy and LAMBADA perplexity: See Tables 1 and 4 | Helped |
| Use RWKVQuant instead of the strongest reported baseline | RWKV vision models · RWKV-T and RWKV-S classification, detection, and segmentation | Top-1 accuracy, Box AP, and MIoU: RWKV-S detection: 44.03 -> 43.95 | Mixed |
| Add hybrid GPTQ and GPTVQ layer selection using the proposed proxies (ablation) | RWKV language models · RWKV7-0.1B through RWKV6-14B | 0-shot 9-task accuracy and LAMBADA perplexity: RWKV7-0.1B: 39.33 -> 40.69 accuracy; 40.16 -> 24.71 PPL | Helped |
| Use the coarse-to-fine proxy instead of alternative proxy measures (ablation) | RWKV7 hybrid quantization · RWKV7-0.1B, RWKV7-0.5B, and RWKV7-1.47B | 0-shot 9-task accuracy and LAMBADA perplexity: Our: 41.04, 45.54, 52.32 accuracy; 19.70, 9.55, 5.24 PPL | Helped |
| Add codebook optimization for element-wise multiplication (ablation) | RWKV models · RWKV7-0.1B through RWKV6-14B | 0-shot 9-task accuracy: RWKV7-0.1B: 40.69 -> 41.09; RWKV6-14B: 62.03 -> 62.69 | Helped |
| Add codebook optimization for element-wise multiplication (ablation) | RWKV models · RWKV7-0.1B through RWKV6-14B | LAMBADA perplexity: RWKV6-7B: 3.21 -> 3.23; RWKV6-14B: 2.89 -> 2.89 | Mixed |
Surprising findings
- Approximately 60% of RWKV layers are judged suitable for scalar quantization, versus about 10% for LLaMA, indicating RWKV weights are more uniform.
- Applying QuaRot to RWKV-7 models increases overall FLOPs by more than 99%, so smoothing/rotation SQ methods carry substantial runtime overhead.
Limitations
- Speedup and memory results are reported only for RWKV6 models on one GPU type.
- Thresholds tau_c and tau_f must be tuned per model, and the reported values vary across settings.
Experimental setup
Datasets: LAMBADA ImageNet COCO ADE20K HEADQA HellaSwag OpenBookQA PIQA SciQ Winogrande ARC-Challenge ARC-Easy
Benchmarks: LAMBADA HEADQA HellaSwag OpenBookQA PIQA SciQ Winogrande ARC-Challenge ARC-Easy ImageNet COCO ADE20K
Metrics: perplexity zero-shot accuracy top-1 accuracy box AP mIoU tokens/sec memory use
Baselines: RTN GPTQ AWQ QuaRot kMeans GPTVQ VPTQ
Builds on: GPTQ GPTVQ VPTQ KMeans
Closest papers by meaning
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.87)
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.87)
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.87)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.86)
- PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.86)
- QBB: Quantization with Binary Bases for LLMs (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.