Research map / Federated learning and privacy / Privacy attacks and secure learning
CipherPrune: Efficient and Scalable Private Transformer Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CipherPrune, a private Transformer inference framework that progressively prunes tokens under encryption and assigns lower-degree polynomial approximations to less important tokens, cutting 2PC runtime by roughly 6.1x at 128 tokens and 10.6x at 512 tokens.
Problem
Private Transformer inference with cryptographic protocols (hybrid HE/MPC) has high latency and scales poorly with long token inputs, since attention cost grows quadratically with token count and high-degree polynomial non-linearities are expensive.
Approach
Input-adaptive, progressive token pruning computed on secret shares (importance scores from attention, comparison against learned thresholds, oblivious swaps to relocate pruned tokens), combined with encrypted polynomial reduction that uses low-degree polynomials for less important tokens, with thresholds learned jointly via a soft-mask gradient search that accounts for crypto cost.
Key results
- On BERT-Large, CipherPrune runs in 157.6 s vs 624.3 s for BOLT (about 3.9x faster) at 92.5% vs 92.4% accuracy.
- CipherPrune achieves about 6.1x speedup over BOLT for 128-token GPT2 inputs and about 10.6x at 512 tokens.
- Communication is reduced by 2.3 to 40.4x relative to prior works across BERT models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CipherPrune instead of BOLT | BERT Medium · BERT Medium | Time (s): 99.5 -> 43.6 | Helped |
| Use CipherPrune instead of BOLT | BERT Base · BERT Base | Time (s): 245.4 -> 79.1 | Helped |
| Use CipherPrune instead of BOLT | BERT Large · BERT Large | Time (s): 624.3 -> 157.6 | Helped |
| Use CipherPrune instead of BOLT | BERT models · BERT Medium, BERT Base, and BERT Large | Communication (GB): 14.3 -> 6.7; 25.7 -> 9.7; 67.9 -> 18.4 | Helped |
| Use CipherPrune instead of IRON | BERT models · BERT Medium, BERT Base, and BERT Large | Accuracy (%): 87.7±0.2 -> 87.4±0.2; 90.4±0.1 -> 90.1±0.2; 92.7±0.1 -> 92.5±0.1 | Hurt |
| Replace fixed 50% word elimination with adaptive, progressive token pruning (ablation) | CipherPrune† vs BOLT with W.E. · NLP tasks | Accuracy (%): +0.5% | Helped |
| Add progressive token pruning to adaptive token pruning (ablation) | CipherPrune† vs BOLT with W.E. · NLP tasks | Runtime speedup: 2.1× | Helped |
| Add polynomial reduction to token pruning (ablation) | CipherPrune · NLP tasks | Runtime speedup over BOLT: 6.1× | Helped |
| Add polynomial reduction to token pruning (ablation) | CipherPrune · NLP tasks | Accuracy | Hurt |
| Increase input length while using CipherPrune rather than BOLT without W.E. | GPT2 · 32 to 512 input tokens | Speedup: ~1.9× -> ~10.6× | Helped |
| Use a large α rather than a smaller α (ablation) | BERT-Base threshold settings · BERT-Base | Accuracy | Helped |
Limitations
- Pruning ratio and polynomial reduction thresholds require offline fine-tuning per model and task.
- Accuracy drops slightly with polynomial reduction compared to the pruning-only variant.
- Semi-honest threat model; the number of pruned tokens is revealed to both parties.
Experimental setup
Datasets: GLUE MNLI GLUE QNLI GLUE SST-2 GLUE MRPC
Benchmarks: GLUE MNLI GLUE QNLI GLUE SST-2 GLUE MRPC
Metrics: runtime (s) communication (GB) accuracy (%)
Baselines: IRON BOLT BOLT w/o W.E. BumbleBee MPCFormer PUMA
Builds on: BOLT IRON EzPC BFV
Closest papers by meaning
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · similarity 0.88)
- An Efficient Private GPT Never Autoregressively Decodes (ICML 2025 · similarity 0.87)
- Jaguar: Fast Private CNN Inference with Power-of-Two Homomorphic Arithmetic (NeurIPS 2026 · similarity 0.87)
- Nimbus: Secure and Efficient Two-Party Inference for Transformers (NeurIPS 2024 · similarity 0.87)
- United We Stand: Accelerating Privacy-Preserving Neural Inference by Conjunctive Optimization with Interleaved Nexus (AAAI 2024 · similarity 0.86)
- HEPrune: Fast Private Training of Deep Neural Networks With Encrypted Data Pruning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.