Research map / Architectures and efficiency / RNNs and normalization
DKM: Differentiable k-Means Clustering Layer for Neural Network Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-dimensional DKM 8/8 instead of DKM 1/1 (ablation) | DKM weight clustering · ImageNet1k, MobileNet-v1 | Top-1 accuracy: 5.9 -> 64.3 | Helped |
| Increase clustering dimension from 4/8 to 8/16 at the same bit-per-weight target (ablation) | DKM weight clustering · ImageNet1k, MobileNet-v2 | Top-1 accuracy: 34.0 -> 58.3 | Helped |
| Increase clustering dimension from 1 to 4 (ablation) | DKM weight clustering · QNLI, MobileBERT | Accuracy: 69.8 -> 78.3 | Helped |
| Use DKM instead of EWGS | Compressed BERT models · QNLI | Accuracy: EWGS 1-bit vs DKM 4/4: ALBERT 62.0 -> 80.0; DistilBERT 60.9 -> 84.0; MobileBERT 60.2 -> 78.3 | Helped |
| Use DKM instead of GOBO | DistilBERT · MNLI | Model size: 23.9 -> 21.8 | Helped |
| Use DKM with 4/1 instead of RPS | GoogleNet · ImageNet1k | Top-1 accuracy: 67.9 -> 70.9 | Helped |
| Use hard assignment instead of DKM soft assignment (ablation) | ResNet18 · ImageNet1k | Top-1 accuracy: over 10% drop | Hurt |
| Use Gumbel-softmax assignment instead of DKM attention assignment (ablation) | ResNet18 · ImageNet1k | Top-1 accuracy: as significant as 3.4% drop | Hurt |
| Snap train-time weights to nearest centroids for inference (ablation) | DKM-compressed vision models · ImageNet1k | Top-1 accuracy: 0.2% - 1.7% | Hurt |
| Vary epsilon from 1e--2 to 1e--7 (ablation) | DKM ResNet18 · ImageNet1k | Top-1 accuracy: 65.7 -65.9% | No clear effect |
| Limit DKM iterations to r<3 for 8/8 clustering (ablation) | DKM ResNet50 · ImageNet1k | Top-1 accuracy: 74.5 -> 74.0 | Hurt |
| Increase tau for harder compression settings (ablation) | DKM weight clustering · ImageNet1k | Top-1 accuracy | Helped |
Closest papers by meaning
- "Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach (NeurIPS 2022 · similarity 0.88)
- Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions (ICML 2018 · similarity 0.86)
- Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering (ICML 2017 · similarity 0.86)
- Forget the Data and Fine-Tuning! Just Fold the Network to Compress (ICLR 2025 · similarity 0.85)
- Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives (ICLR 2025 · similarity 0.84)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.