Research map / Federated learning and privacy / Privacy attacks and secure learning
An Efficient Private GPT Never Autoregressively Decodes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes POST, a secure GPT decoding scheme where a client-side public model drafts tokens that a server-side private model verifies via a secure speculative sampling protocol, with distillation-based alignment, yielding 2.1x-6.0x speedups.
Problem
Secure two-party GPT inference via HE and MPC is far too slow, especially during autoregressive decoding where each step forwards only one token.
Approach
Secure forward latency is nearly insensitive to input length, so the client drafts several tokens with a public GPT and the private model verifies them all in one secure step using a privately executed speculative sampling protocol; knowledge distillation aligns the public model to raise acceptance.
Key results
- After distillation alignment, acceptance ratios with 8 draft tokens range from 52% to 84% across model pairs and tasks.
- End-to-end speedup over standard secure decoding ranges from 2.1x to 6.0x across three public-private model pairs and two network conditions (LAN 1Gbps/10ms, WAN 400Mbps/40ms).
- Optimized secure speculative sampling takes about 1.19-4.12 s versus 14.78-46.62 s for the naive version, roughly a tenfold reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add knowledge distillation alignment to LLaMA-68M (ablation) | POST with Vicuna-7B · Spider | acceptance ratio: 0.240 -> 0.614 | Helped |
| Add knowledge distillation alignment to LLaMA-68M (ablation) | POST with Vicuna-7B · GSM8K | acceptance ratio: 0.462 -> 0.662 | Helped |
| Add knowledge distillation alignment to LLaMA-68M (ablation) | POST with Vicuna-7B · Code-search-Python | acceptance ratio: 0.366 -> 0.561 | Helped |
| Add knowledge distillation alignment to LLaMA-68M (ablation) | POST with Vicuna-7B · Alpaca-finance | acceptance ratio: 0.512 -> 0.626 | Helped |
| Optimize secure speculative sampling | POST with Vicuna-7B · LAN (1 Gbps, 10 ms), draft length 4 | sampling time: 14.78 -> 1.19 s | Helped |
| Use public decoding with secure verification instead of standard secure decoding | Secure GPT decoding · Three public-private model pairs, two network conditions | speedup: 2.1× to 6.0× | Helped |
Surprising findings
- Secure decoding latency is nearly insensitive to input length, so verifying several draft tokens costs about the same as decoding one token.
Limitations
- Speedup depends on acceptance ratio, which is limited for public models that differ in training data from the private model.
- Alignment with top-K (about 5) distributions only, as commercial APIs return limited probabilities.
- Experiments use small public models and secure evaluation only on two nodes with 64 vCPUs.
Experimental setup
Datasets: Spider GSM8K Code-search-Python Alpaca-finance
Benchmarks: Spider GSM8K Code-search-Python Alpaca-finance
Metrics: acceptance ratio end-to-end speedup decoder latency (s)
Baselines: BumbleBee Nimbus standard secure decoding
Builds on: speculative sampling speculative decoding BumbleBee Nimbus SecretFlow-SPU
Closest papers by meaning
- CipherPrune: Efficient and Scalable Private Transformer Inference (ICLR 2025 · similarity 0.87)
- EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption (ICML 2025 · similarity 0.86)
- Encryption-Friendly LLM Architecture (ICLR 2025 · similarity 0.85)
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · similarity 0.85)
- DCT-CryptoNets: Scaling Private Inference in the Frequency Domain (ICLR 2025 · similarity 0.85)
- Ditto: Quantization-aware Secure Inference of Transformers upon MPC (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.