Research map / Architectures and efficiency / RNNs and normalization
StrassenNets: Deep Learning with a Multiplication Budget
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply learned SPN matrix multiplication in place of standard layer multiplications | ResNet-18 · ImageNet validation | multiplication reduction (%): 99.63 | Helped |
| add knowledge distillation (ablation) | ST-ResNet-18 (r=2c_out, p=2, g=1) · ImageNet validation | top-1 accuracy (%): 67.1 -> 69.4 | Helped |
| add knowledge distillation (ablation) | ST-LM (r=n_out) · PTB test | perplexity: 83.42 -> 79.11 | Helped |
| increase spatial patch size p from 1 to 2 (ablation) | ST-ResNet-20 (r=c_out) · CIFAR-10 test | accuracy (%): 91.24 -> 89.87 | Hurt |
| increase spatial patch size p from 1 to 2 (ablation) | ST-ResNet-18 (r=2c_out, g=1) · ImageNet validation | top-1 accuracy (%): 67.0 -> 67.1 | Helped |
| use grouped convolution with g=4 rather than g=1 (ablation) | ST-ResNet-18 (r=2c_out, p=1) · ImageNet validation | top-1 accuracy (%): 67.0 -> 65.6 | Hurt |
| reduce multiplication budget r from 6c_out to 4c_out (ablation) | ST-ResNet-18 (p=2, g=1) · ImageNet validation | top-1 accuracy (%): 68.2 -> 68.0 | Hurt |
| reduce r from c_out to 3/4 c_out (ablation) | ST-ResNet-20 (p=1) · CIFAR-10 test | accuracy (%): 91.24 -> 90.62 | Hurt |
| learn a per-layer multiplication allocation using L1 regularization on a_tilde rather than fixing r/c_out (ablation) | ST-ResNet · ImageNet validation | accuracy | Hurt |
| pretrain full-precision SPN weights before activating quantization (ablation) | StrassenNets · DNN training | accuracy | No clear effect |
| use ST-LM rather than TWN quantization at a similar multiplication budget | language model · PTB test | perplexity: 92.23 -> 83.42 | Helped |
| use KD with TWN quantization (ablation) | TWN language model · PTB test | perplexity | Hurt |
Closest papers by meaning
- An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks (ICML 2025 · similarity 0.87)
- Multiplication-Free Transformer Training via Piecewise Affine Operations (NeurIPS 2023 · similarity 0.87)
- ShiftAddNet: A Hardware-Inspired Deep Network (NeurIPS 2020 · similarity 0.86)
- ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks (ICML 2022 · similarity 0.86)
- Convolutional Rectifier Networks as Generalized Tensor Decompositions (ICML 2016 · similarity 0.86)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.