Research map / Architectures and efficiency / Pruning and efficient tuning
In defense of parameter sharing for model-compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows randomized parameter sharing (RPS) beats pruning and smaller models at high compression, and fixes ROAST's stability and Pareto-continuity issues via STABLE-RPS (gradient scaling plus RANDOMFOLD mapping).
Problem
Reducing a model's parameter memory footprint at the start of training: whether pruning or randomized parameter sharing (RPS) gives a better memory-accuracy tradeoff, and ROAST's instability and lack of Pareto-continuity.
Approach
Proposes STABLE-RPS, which adds a gradient scaler (theory-driven or effective-update) to remove ROAST's sensitivity to init-stdev, and a RANDOMFOLD mapping with optimal load factor so that compression 1x recovers the full model; it argues parameter sharing beats pruning in capacity.
Key results
- STABLE-RPS keeps CIFAR-10 ResNet-20 accuracy near 91% across init-stdev from 1e-3 to 10 at 1.33x compression, while ROAST ranges from 10.0% to 91.5%.
- Uninformed RPS outperforms or matches moderately informed pruning (MAG, SNIP, SYNFLOW, GRASP) across compression and beats LTR at high compression, per the paper's experiments on over 1300 runs.
- Theory shows the expected variance of norm/inner-product estimation under STABLE-RPS is no larger than under pruning for i.i.d. Gaussian inputs with unequal variances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STABLE-RPS gradient scaling instead of ROAST's unscaled updates (ablation) | ROAST · CIFAR-10 (ResNet-20), across compression levels | accuracy stability across init-stdev: ROAST: 10.0 to 91.5; STABLE-RPS: 91.4 to 91.2 (1.33×); ROAST: 10.0 to 87.2; STABLE-RPS: 87.3 to 87.4 (10×); ROAST: 10.0 | Helped |
| Use STABLE-RPS gradient scaling instead of ROAST's unscaled updates (ablation) | ROAST · CIFAR-100 (VGG-11), across compression levels | accuracy stability across init-stdev: ROAST: 1.0 to 67.5; STABLE-RPS: 68.1 to 68.9 (1.33×); ROAST: 38.7 to 66.0; STABLE-RPS: 65.8 to 66.5 (10×); ROAST: 1.0 to | Helped |
| Replace ROAST mapping with RANDOMFOLD mapping (ablation) | ROAST · CIFAR-10 and CIFAR-100 vision tasks, across compression levels | accuracy | Helped |
| Use RANDOMFOLD mapping rather than element-wise or ROAST mapping at full model memory (ablation) | RPS · Vision tasks at full model memory | accuracy | Helped |
| Use STABLE-RPS parameter sharing rather than RAND pruning | Model compression · Vision tasks, all compression levels | accuracy | Helped |
| Use STABLE-RPS parameter sharing rather than moderately informed pruning | Model compression · Vision tasks, especially higher compression | accuracy | Helped |
| Use STABLE-RPS rather than Lottery Ticket Rewinding | Model compression · Vision tasks, high compression | accuracy | Helped |
| Use STABLE-RPS rather than a smaller model with reduced hidden dimensions | Model compression · Vision tasks | accuracy | Helped |
| Increase ROAST init-stdev from 0.1 to 10 (ablation) | ROAST · CIFAR-10 (ResNet-20) and CIFAR-100 (VGG-11), across compression levels | accuracy: CIFAR-10: 91.5 to 79.4 (1.33×), 87.2 to 77.3 (10×), 75.5 to 66.8 (100×), 53.1 to 52.1 (1000×); CIFAR-100: 67.5 to 19.8 ( | Hurt |
Surprising findings
- Lottery Ticket Rewinding experiences layer collapse at high compression, sometimes performing worse than moderately informed pruning.
- Uninformed parameter sharing beats even the highly informed LTR pruning at high compression.
Limitations
- Parameter sharing requires the same computation as the full architecture, so no compute savings.
- Only post-training-free (start-of-training) compression is studied; no post-training parameter-sharing scheme.
- Theory covers linear models and norm preservation under i.i.d. assumptions, not deep nets directly.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: accuracy compression factor norm of solution
Baselines: RAND MAG SNIP SYNFLOW GRASP LTR smaller model ROAST
Builds on: ROAST HashedNet ROBE Lottery Ticket Hypothesis
Closest papers by meaning
- DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models (ICLR 2025 · similarity 0.87)
- Forget the Data and Fine-Tuning! Just Fold the Network to Compress (ICLR 2025 · similarity 0.87)
- You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning (ICLR 2025 · similarity 0.86)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.86)
- Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression (NeurIPS 2025 · similarity 0.86)
- Unified Scaling Laws for Compressed Representations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.