Research map / Architectures and efficiency / RNNs and normalization
Scaling Up Exact Neural Network Compression by ReLU Stability
Thiago Serra, Xin Yu, Abhinav Kumar, Srikumar Ramalingam
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| solve a single MILP for all neurons (ISA) instead of one per neuron | exact compression · MNIST | median runtime: 83 times faster | Improved |
| single MILP instead of per-neuron MILP | exact compression · CIFAR-10 | median runtime: 183 times faster | Improved |
| single MILP instead of per-neuron MILP | exact compression · CIFAR-100 | median runtime: 137 times faster | Improved |
| increase network size from 2x100 to 2x400 | ISA vs baseline · MNIST | median runtime ratio: 77 -> 193 | Improved |
| add GPU-based preprocessing step (ablation) | ISA · MNIST 2x100 | median runtime: 3.2 times speed up | Improved |
| add moderate L1 regularization | ReLU MLP · MNIST 2x100 | accuracy: 97.92 -> 98.13 | Improved |
| add strong L1 regularization (0.0004) | ReLU MLP · MNIST 2x100 | accuracy: 97.92 -> 97.28 | Hurt |
| add L1 regularization | ReLU MLP · MNIST 4x100 | accuracy: 97.93 -> 98.23 | Improved |
| increase L1 regularization from 0 to 0.0002 | LEO++ · MNIST 2x100 | % neurons removed: 0 -> 18 | Improved |
| increase depth from 2x100 to 5x100 at l1=0 | ISA + LEO++ · MNIST | compression runtime (s): 3.4 -> 2000 | Hurt |
| use preprocessing alone to decide stability (inexact) | ISA · MNIST | % extra neurons removed: 1.16 | No clear effect |
| constrain sum of MNIST inputs to [15, 320] instead of [0, 784] (ablation) | ISA · MNIST | runtime: 17.7% lower geometric mean | Improved |
Closest papers by meaning
- Convergence of Near-Linear Width ReLU Networks with Unbalanced Initialization (NeurIPS 2026 · similarity 0.86)
- On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks (ICLR 2019 · similarity 0.86)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.85)
- Expressivity of ReLU-Networks under Convex Relaxations (ICLR 2024 · similarity 0.85)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.85)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
