Research map / Architectures and efficiency / Quantization
Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JLCM, a codebook-based DNN weight compression method that groups neurons by distribution, uses per-group scales or multiple codebooks without mapping overhead, and jointly learns codebooks and mappings with a proximal straight-through-style gradient.
Problem
Codebook-based memory compression of DNN weights suffers from the granularity problem (single codebook per tensor or costly multi-codebook mappings) and from gradient-based mapping optimization that favors jumps to extreme codebook values instead of proximal ones, preventing joint learning of codebo
Approach
Reorder neurons by weight-distribution clustering so that the row index implicitly selects a codebook or scale group (no extra mapping cost), then jointly learn codebooks and soft mappings layer-by-layer with a redefined gradient for the mapping that decreases with codeword distance, favoring proximal updates.
Key results
- On ResNet 18 at 5.33 compression, data-driven JLCM reaches 62.939% ImageNet accuracy vs 53.000% for NUPES (previous best), and data-free JLCM init reaches 41.133% vs 19.532% for SQuant.
- On Llama 7B, JLCM at 5.33 compression retains 55.940 average common-sense score vs 56.140 original, outperforming OPTQ (49.938 at 5.33) and reaching 53.081 at 7.0 compression.
- On Stable Diffusion 2.0, data-free JLCM init at 3.9 compression reaches CLIP score 28.471 vs original 28.5353 and 27.498 for RED++.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn the codebook only (ablation) | JLCM · ImageNet, ResNet18, α=7.5 | accuracy: 20.555 -> 22.149 | Helped |
| learn the codebook only (ablation) | JLCM · ImageNet, ViT b16, α=7.5 | accuracy: 60.691 -> 63.912 | Helped |
| jointly learn codebook and mappings with the standard gradient update (ablation) | JLCM · ImageNet, ResNet18, α=7.5 | accuracy: 22.149 -> 20.343 | Hurt |
| jointly learn codebook and mappings with the standard gradient update (ablation) | JLCM · ImageNet, ViT b16, α=7.5 | accuracy: 63.912 -> 57.800 | Hurt |
| use proximal search for joint codebook and mapping learning (ablation) | JLCM · ImageNet, ResNet18, α=7.5 | accuracy: 20.343 -> 41.779 | Helped |
| use proximal search for joint codebook and mapping learning (ablation) | JLCM · ImageNet, ViT b16, α=7.5 | accuracy: 57.800 -> 69.996 | Helped |
| use JLCM instead of NUPES at compression rate 5.33 | ImageNet ResNet 18 · ImageNet, ResNet 18, α=5.33 | accuracy: 53.000 -> 62.939 | Helped |
| use JLCM instead of NUPES at compression rate 5.33 | ImageNet ViT b16 · ImageNet, ViT b16, α=5.33 | accuracy: 77.231 -> 80.558 | Helped |
| use JLCM initialization instead of RED++ | Stable Diffusion 2.0 · DiffusionDB | CLIP Score: 27.498 -> 28.471 | Helped |
| use JLCM instead of OPTQ at compression rate 5.33 | Llama 7B · Common sense reasoning | accuracy: 49.938 -> 55.940 | Helped |
Surprising findings
- Naive joint learning of codebooks and mappings with standard chain-rule gradients degrades accuracy (ResNet18 20.343 vs 22.149 when only the codebook is learned), so joint optimization only helps with the proposed proximal gradient.
- Random clustering for initialization yields near-zero accuracy on all tested networks, showing initialization quality is critical.
Limitations
- Joint learning with standard gradients fails, relying on a heuristic gradient redefinition without full theoretical grounding.
- Experiments only cover ResNet 18, MobileNet v2, EfficientNet B0, ViT b16, Llama 7B and Stable Diffusion 2.0; the 2GB mobile claim is not measured on devices.
- Gains at high compression (7.5) on ResNet 18 are much lower than at 5.33 (41.779 vs 62.939 with JLCM).
Experimental setup
Datasets: ImageNet DiffusionDB common sense reasoning benchmarks (calibration/evaluation)
Benchmarks: ImageNet DiffusionDB common sense reasoning tasks
Metrics: top-1 accuracy CLIP score average accuracy on common sense reasoning compression rate
Baselines: DFQ SQuant PowerQuant RED++ AdaRound BRECQ NUPES OPTQ LLM.int8()
Builds on: AdaRound BRECQ straight-through estimation OPTQ
Closest papers by meaning
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks (ICLR 2020 · similarity 0.90)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.89)
- Model compression via distillation and quantization (ICLR 2018 · similarity 0.89)
- Focused Quantization for Sparse CNNs (NeurIPS 2019 · similarity 0.88)
- Quantization without Tears (CVPR 2025 · similarity 0.88)
- Qinco2: Vector Compression and Search with Improved Implicit Neural Codebooks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.