Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Transfer Learning Accelerates and Enhances Certified Robustness: A Comprehensive Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Accelerates randomized-smoothing L2 certification for downstream tasks by reusing pre-training verification results and applying sparse transfer-learning updates, with DiffStab regularizer and RigL dynamic masks to recover certified accuracy.
Problem
Randomized-smoothing L2 certified robustness verification is computationally expensive (about 1,000 noisy forward passes per sample), and the cost grows with model size.
Approach
Reuse dense verification results from a pre-trained network and compute only the differential forward pass caused by sparse weight updates during transfer (unstructured, channel-structured, or group-wise masks), then regularize with a neuron-bound stability loss (DiffStab) and dynamic mask selection via RigL to close the certified-accuracy gap to dense transfer.
Key results
- Sparse differential verification on CIFAR10 (ResNet-50, structured 8% updated params) cuts verification time by 72.8% with verified accuracy only about 1% below direct training after DiffStab+RigL.
- On CelebV-HQ, group-wise sparsity at 16% updated parameters gives under 1% drop versus dense transfer while saving 49.3% verification time.
- Few-shot CelebV-HQ: unstructured and group-wise sparse configs updating 2% of parameters lose under 2% verified accuracy with 63.5% and 81.8% time savings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unstructured sparse transfer instead of dense transfer (ablation) | ResNet-50 · CIFAR10 | verified accuracy: 61.2 -> 55.1 (1% updated); 61.2 -> 61.1 (64% updated) | Mixed |
| Use structured sparse transfer instead of dense transfer (ablation) | ResNet-50 · CIFAR10 | verification time saved: 92.1 | Helped |
| Use structured sparse transfer instead of dense transfer (ablation) | ResNet-50 · CIFAR10 | verified accuracy: 61.2 -> 53.4 (1% updated) | Hurt |
| Use group-wise sparse transfer instead of dense transfer (ablation) | ResNet-50 · CIFAR10 | verified accuracy: 61.2 -> 54.0 (1% updated); 61.2 -> 60.5 (64% updated) | Mixed |
| Add DiffStab and RigL to unstructured sparse transfer (ablation) | ResNet-50 sparse transfer · CIFAR10 | verified accuracy: 55.1 -> 58.1 (+3.0) | Helped |
| Add DiffStab and RigL to structured sparse transfer (ablation) | ResNet-50 sparse transfer · CIFAR10 | verified accuracy: 57.6 -> 60.6 (+3.0) | Helped |
| Add DiffStab and RigL to group-wise sparse transfer (ablation) | ResNet-50 sparse transfer · CIFAR10 | verified accuracy: 60.5 -> 61.7 (+1.2) | Helped |
| Add DiffStab and RigL to unstructured sparse transfer (ablation) | ResNet-50 sparse transfer · CelebV-HQ, normal transfer | verified accuracy: 52.8 -> 56.2 (+3.4) | Helped |
| Add DiffStab and RigL to structured sparse transfer (ablation) | ResNet-50 sparse transfer · CelebV-HQ, normal transfer | verified accuracy: 53.8 -> 56.3 (+2.5) | Helped |
| Add DiffStab and RigL to group-wise sparse transfer (ablation) | ResNet-50 sparse transfer · CelebV-HQ, normal transfer | verified accuracy: 54.2 -> 57.2 (+3.0) | Helped |
| Add DiffStab and RigL to unstructured sparse transfer (ablation) | ResNet-50 sparse transfer · CelebV-HQ, few-shot transfer | verified accuracy: 48.6 -> 52.5 (+3.9) | Helped |
| Add DiffStab and RigL to structured sparse transfer (ablation) | ResNet-50 sparse transfer · CelebV-HQ, few-shot transfer | verified accuracy: 50.2 -> 51.9 (+1.7) | Helped |
Surprising findings
- Structured sparsity gives the largest verification speedup but the lowest verified accuracy, the reverse ordering of acceleration effects.
- Sparse transfer without constraints did not beat dense transfer in certified robustness, motivating the stability regularizer.
Limitations
- Requires identical inputs between pre-training and downstream tasks to reuse verification results.
- Speedup is small for very sparse ratios in some architectures; VGG-16 and ResNet gains vary with sparsity pattern.
- Evaluated only on CIFAR10 and CelebV-HQ with ResNet-style and VGG models at small scale.
Experimental setup
Datasets: CIFAR10 CelebV-HQ
Benchmarks: CIFAR10 CelebV-HQ
Metrics: verified accuracy verification time saved (%)
Baselines: Direct Train Dense Transfer Unstructured sparse transfer (baseline)
Builds on: randomized smoothing DiffPruning RigL Center Smoothing Interval Bound Propagation
Closest papers by meaning
- Halt Fast! Early Stopping for Certified Robustness (NeurIPS 2026 · similarity 0.88)
- Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing (NeurIPS 2024 · similarity 0.86)
- Probably Approximately Global Robustness Certification (ICML 2025 · similarity 0.85)
- Mitigating the Curse of Dimensionality for Certified Robustness via Dual Randomized Smoothing (ICLR 2024 · similarity 0.85)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.85)
- Towards Large Certified Radius in Randomized Smoothing Using Quasiconcave Optimization (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.