Research map / Architectures and efficiency / Quantization
S$^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase training epochs from 90 to 200 (ablation) | S3 3-bit shift ResNet-18 · ImageNet val, ResNet-18 | Top-1 accuracy: 68.45 -> 69.83 | Helped |
| increase training epochs from 90 to 150 (ablation) | S3 3-bit shift ResNet-50 · ImageNet val, ResNet-50 | Top-1 accuracy: 74.67 -> 75.76 | Helped |
| train 200 vs 150 epochs (ablation) | S3 3-bit shift ResNet-50 · ImageNet val, ResNet-50 | Top-1 accuracy: 75.76 -> 75.75 | No clear effect |
| replace staircase-like quantizer with S3 power-of-two parameterization (ablation) | 3-bit shift network · ImageNet val, ResNet-18 | Top-1 accuracy: 68.59 -> 69.82 | Helped |
| vary dense weight regularizer alpha from 1e-2 to 1e-6, no decay (ablation) | S3 3-bit shift · ImageNet val, ResNet-18 | Top-1 accuracy: 69.78 to 69.85 | No clear effect |
| add linear or cosine decay to alpha with alpha=1e-6 (ablation) | S3 3-bit shift · ImageNet val, ResNet-18 | Top-1 accuracy: 69.83 -> 66.70 | Hurt |
| add decay scheduler to alpha at larger values (ablation) | S3 3-bit shift · ImageNet val, ResNet-18 | Top-1 accuracy: 69.82 -> 69.91 | No clear effect |
| remove dense weight regularizer (ablation) | S3 reparameterized network · ImageNet val, ResNet-18 | training convergence | Hurt |
| S3 reparameterization of ternary weights vs standard quantizer | ternary ResNet-20 · CIFAR-10 | weight dynamics (WSVR, WLVR) | Helped |
| S3 3-bit shift random init vs DeepShift 6-bit pre-trained | ResNet-18 · ImageNet val, ResNet-18 | Top-1 accuracy: 68.32 -> 69.82 | Helped |
| S3 3-bit shift random init vs INQ 5-bit pre-trained | ResNet-18 · ImageNet val, ResNet-18 | Top-1 accuracy: 68.98 -> 69.82 | Helped |
| S3 3-bit shift random init vs DeepShift 6-bit random init | ResNet-18 · ImageNet val, ResNet-18 | Top-1 accuracy: 65.63 -> 69.82 | Helped |
Closest papers by meaning
- All You Need Is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification (CVPR 2019 · similarity 0.85)
- Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression (ICML 2026 · similarity 0.84)
- Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · similarity 0.83)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.83)
- AMPA: Adaptive Mixed Precision Allocation for Low-Bit Integer Training (ICML 2024 · similarity 0.83)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.