Research map / Architectures and efficiency / Pruning and efficient tuning
A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves a rescaling-invariant parameter-space Lipschitz bound for ReLU DAG networks using the l1 path-metric of path-lifting, and derives a rescaling-invariant path-magnitude pruning criterion tested on ResNet-18/ImageNet.
Problem
Parameter-space Lipschitz bounds for ReLU networks are not invariant to neuron-wise rescaling symmetries and only cover plain layered MLPs, limiting their use for pruning, quantization and generalization guarantees.
Approach
Bounds the change in network output under weight perturbations by the l1 distance between path-lifting vectors (products of weights along paths), which is invariant to neuron-wise rescaling, for general ReLU DAGs with pooling, skip connections and frozen batch norm; under same-sign assumptions, and computable in two forward passes under practical conditions. Uses the resulting per-weight path-magn
Key results
- Path-magnitude pruning on ResNet-18/ImageNet at 40/60/80% sparsity gives 68.6/67.9/66.0% top-1 vs magnitude 68.8/68.2/66.5%.
- After random neuron-wise rescaling, magnitude pruning drops to 63.1/57.5/15.8% while path-magnitude is unchanged.
- Path-magnitude scores for all weights of ResNet-18 are computed in about 32 ms, comparable to a forward pass.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use path-magnitude instead of magnitude pruning | ResNet-18 · ImageNet-1k, 40% pruning | top-1 accuracy: 68.8 vs 68.6 | No clear effect |
| use path-magnitude instead of magnitude pruning | ResNet-18 · ImageNet-1k, 60% pruning | top-1 accuracy: 68.2 vs 67.9 | No clear effect |
| use path-magnitude instead of magnitude pruning | ResNet-18 · ImageNet-1k, 80% pruning | top-1 accuracy: 66.5 vs 66.0 | No clear effect |
| apply random neuron-wise rescaling before magnitude pruning | ResNet-18 · ImageNet-1k, 40% pruning | top-1 accuracy: 68.8 -> 63.1 | Hurt |
| apply random neuron-wise rescaling before magnitude pruning | ResNet-18 · ImageNet-1k, 60% pruning | top-1 accuracy: 68.2 -> 57.5 | Hurt |
| apply random neuron-wise rescaling before magnitude pruning | ResNet-18 · ImageNet-1k, 80% pruning | top-1 accuracy: 66.5 -> 15.8 | Hurt |
| apply random neuron-wise rescaling before path-magnitude pruning | ResNet-18 · ImageNet-1k, 40% pruning | top-1 accuracy | No clear effect |
Surprising findings
- Lipschitz bounds in parameter space can be made arbitrarily pessimistic under rescaling; a sign-change impossibility shows no finite constant can hold when signs flip.
Limitations
- Bound requires same-sign parameters (theta_i * theta'_i >= 0); sign flips give no finite constant.
- Pruning is a proof-of-concept using one-shot per-weight scores without accounting for global dependencies; no comparison with stronger methods such as CHITA.
- Rademacher generalization bounds derived from it are looser by roughly sqrt(#params) than prior path-norm analyses.
Experimental setup
Datasets: ImageNet-1k
Benchmarks: ImageNet-1k
Metrics: top-1 accuracy run-time (ms)
Baselines: magnitude pruning magnitude pruning after random rescaling OBD
Builds on: path-lifting path-norm lottery ticket hypothesis
Closest papers by meaning
- Approximation theory for 1-Lipschitz ResNets (NeurIPS 2025 · similarity 0.89)
- A path-norm toolkit for modern networks: consequences, promises and challenges (ICLR 2024 · similarity 0.89)
- Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks (NeurIPS 2019 · similarity 0.89)
- Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations (ICLR 2024 · similarity 0.88)
- Characterizing ResNet's Universal Approximation Capability (ICML 2024 · similarity 0.88)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.