Research map / Architectures and efficiency / Neural architecture search
Seesaw: Compensating for Nonlinear Reduction with Linear Computations for Private Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Seesaw is a one-shot NAS method for private inference that adds parallel linear branches and ReLU-output reuse via ADD-based residual aggregation to compensate for pruned nonlinear operators, with latency-aware loss.
Problem
Reducing expensive nonlinear (ReLU) operations in cryptographic private inference (PPML) hurts model accuracy, creating a tradeoff between accuracy and online/offline latency.
Approach
Compensate for ReLU reduction by adding many linear branches in sampling blocks and reusing ReLU outputs via residual ADD-based aggregation blocks; search jointly trains supermodel and architecture parameters with a latency-aware loss and prunes linear branches to balance online/offline latency.
Key results
- On ImageNet, Seesaw achieves 1.68× lower online latency than SENet at 71% accuracy and 1.55× lower end-to-end latency.
- At iso-latency of 190 seconds on ImageNet, Seesaw reaches 74.81% accuracy, 3.65% better than SENet.
- On CIFAR100, Seesaw achieves 1.53× online latency reduction than SENet at 70% iso-accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Seesaw instead of SENet at iso-accuracy 71% | ImageNet inference · ImageNet | online latency: 1.68× lower | Helped |
| use Seesaw instead of SENet at iso-accuracy 71% | ImageNet inference · ImageNet | online + offline latency: 1.55× lower | Helped |
| use Seesaw instead of SENet at iso-latency | ImageNet inference · ImageNet | accuracy: 3.65% higher | Helped |
| use Seesaw instead of SENet at iso-accuracy 70% | CIFAR100 inference · CIFAR100 | online latency: 1.53× faster | Helped |
| use Seesaw instead of SENet at iso-latency | CIFAR100 inference · CIFAR100 | accuracy: 0.25% higher | Helped |
| add pixel-level SNL pruning (ablation) | Seesaw on CIFAR100 · CIFAR100, ReLU budget 25K | accuracy: 70.59% -> 71.42% | Helped |
| add pixel-level SNL pruning (ablation) | Seesaw on CIFAR100 · CIFAR100, ReLU budget 50K | accuracy: 75.28% -> 76.02% | Helped |
| use ADD instead of CONCAT in sampling blocks (ablation) | Seesaw · CIFAR100 | accuracy: 7.0% on average | Helped |
| prune linear branches using the Seesaw learned-weight method instead of keeping all (ablation) | Seesaw · CIFAR100, multiple ReLU budgets | end-to-end latency | Helped |
| keep a fixed 11 branches per block instead of adaptive pruning (ablation) | Seesaw · CIFAR100, ReLU budget 110K | accuracy: 2.03% accuracy degradation | Hurt |
| keep a fixed 11 branches per block instead of adaptive pruning (ablation) | Seesaw · CIFAR100, ReLU budgets 170K and 300K | accuracy | Hurt |
| keep a fixed 11 branches per block instead of adaptive pruning (ablation) | Seesaw · CIFAR100, ReLU budget 36K | end-to-end latency: 39% slowdown | Hurt |
Surprising findings
- Using CONCAT instead of ADD for aggregation costs about 7.0% accuracy on average at equal ReLU budgets.
- Keeping all linear branches without pruning gives 2x-5x higher latency and sometimes lower accuracy than pruned models.
Limitations
- Offline pre-processing cost of extra linear operations must be controlled via pruning.
- At high ReLU budgets on CIFAR100, gains over SENet are small or negative due to SENet's pixel-level pruning.
Experimental setup
Datasets: ImageNet CIFAR100
Benchmarks: ImageNet CIFAR100
Metrics: top-1 accuracy online latency (seconds) end-to-end online+offline latency (seconds)
Baselines: DELPHI Sphynx SNL SENet ResNet-18 ResNet-34
Builds on: DELPHI ProxylessNAS SENet CryptoNAS SNL
Closest papers by meaning
- United We Stand: Accelerating Privacy-Preserving Neural Inference by Conjunctive Optimization with Interleaved Nexus (AAAI 2024 · similarity 0.84)
- ViP: A Differentially Private Foundation Model for Computer Vision (ICML 2024 · similarity 0.82)
- PrivCirNet: Efficient Private Inference via Block Circulant Transformation (NeurIPS 2024 · similarity 0.82)
- Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training (NeurIPS 2024 · similarity 0.82)
- A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization (ICML 2024 · similarity 0.82)
- MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.