Research map / Label-efficient and robust learning / Knowledge distillation
Light Multi-Segment Activation for Model Compression
Zhenhui Xu, Guolin Ke, Jia Zhang, Jiang Bian, Tie-Yan Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ReLU with LMA-8 | Distilled compact image classifiers · CIFAR-10, Students 1–3 | accuracy: 88.74 $\pm$0.25 -> 90.57 $\pm$0.20; 82.67 $\pm$0.46 -> 85.66 $\pm$0.34; 73.33 $\pm$0.79 -> 77.66 $\pm$0.47 | Improved |
| Replace ReLU with LMA-8 | Distilled WRN-10 students · CIFAR-100 | accuracy: 69.11 $\pm$0.80 -> 70.92 $\pm$0.42; 63.12 $\pm$1.00 -> 66.31 $\pm$0.68 | Improved |
| Replace ReLU with LMA-8 | Distilled translation models · Ope, Students 1–3 | BLEU: 13.73 $\pm$0.19 -> 14.11 $\pm$0.12; 10.46 $\pm$0.18 -> 10.94 $\pm$0.18; ~~6.12 $\pm$0.12 -> ~~6.64 $\pm$0.04 | Improved |
| Replace ReLU with LMA-8 | Distilled translation models · WMT13 | BLEU: 26.89 $\pm$0.05 -> 27.12 $\pm$0.07; 20.39 $\pm$0.09 -> 21.19 $\pm$0.08 | Improved |
| Replace ReLU with LMA-8 | Distilled translation models · Ope and WMT13 | perplexity: 31.84 $\pm$0.31 -> 30.21 $\pm$0.25; 44.51 $\pm$0.52 -> 41.21 $\pm$0.35; 71.69 $\pm$0.51 -> 67.62 $\pm$0.31; ~~6.44 $\pm$ | Improved |
| Use LMA-8 rather than ReLU in quantized distillation | Quantized CIFAR-10 student models · CIFAR-10, 4 bits | accuracy: 85.74 $\pm$0.15 -> 86.31 $\pm$0.41; 77.04 $\pm$0.51 -> 79.48 $\pm$0.79; 65.33 $\pm$0.17 -> 68.85 $\pm$0.99 | Improved |
| Use LMA-8 rather than ReLU in quantized distillation | Quantized CIFAR-10 student models · CIFAR-10, 8 bits | accuracy: 87.02 $\pm$0.23 -> 88.56 $\pm$0.52; 80.53 $\pm$0.75 -> 83.37 $\pm$0.51; 70.23 $\pm$0.98 -> 74.47 $\pm$0.74 | Improved |
| Replace ReLU with LMA-8 instead of APLU-8 | Compact models · CIFAR-10 and translation tasks | inference memory | Improved |
| Increase the LMA segment count from 4 to 8 (ablation) | CIFAR-10 student models · CIFAR-10 | accuracy | Improved |
| Increase the LMA segment count beyond 10 (ablation) | CIFAR-10 student models · CIFAR-10 | accuracy | Hurt |
| Replace LMA with Maxout-8 at the same Student 1 model size (ablation) | CIFAR-10 Student 1 · CIFAR-10 | accuracy: 88.74$\pm$0.25% -> 87.76$\pm$0.55% | Hurt |
Closest papers by meaning
- Model compression via distillation and quantization (ICLR 2018 · similarity 0.87)
- Progressive Network Grafting for Few-Shot Knowledge Distillation (AAAI 2021 · similarity 0.87)
- Few Sample Knowledge Distillation for Efficient Network Compression (CVPR 2020 · similarity 0.87)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.86)
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.85)
- Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
