Research map / Architectures and efficiency / Quantization
Ditto: Quantization-aware Secure Inference of Transformers upon MPC
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ditto combines static dyadic (fixed-point) quantization with quantization-aware distillation and new MPC type-conversion primitives (upcast/downcast across rings) to speed up 3PC secure Transformer inference for Bert and GPT2.
Problem
Secure (MPC-based) Transformer inference is costly in computation and communication, and plaintext quantization (scaling, clipping, dynamic min/max) does not translate efficiently to MPC, with type conversions between rings being non-trivial.
Approach
Use MPC-friendly static dyadic fixed-point quantization with layer-wise precision, recover accuracy via layer-wise quantization-aware distillation, and implement secure upcast/downcast primitives so that different fixed-point types live in different RSS rings during inference.
Key results
- Ditto is about 3.14~4.40x faster than MPCFormer (Quad) and 1.44~2.35x faster than PUMA on Bert and GPT2 in LAN with negligible utility drop.
- On Bert-base, Ditto (Quad) reaches RTE 67.82, CoLA 54.52, QQP 87.72, QNLI 91.78 versus baseline 68.59, 57.06, 87.96, 91.62.
- Ditto's communication is 2.37~3.43x lower than PUMA and 1.28~1.70x lower than MPCFormer (Quad) on the four models, though GPT2-base communication is higher than MPCFormer's.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ditto quantization and Quad GeLU approximation instead of the vanilla baseline | Bert-base · RTE | accuracy: 68.59 -> 67.82 | Hurt |
| Use Ditto quantization and Quad GeLU approximation instead of the vanilla baseline | Bert-base · CoLA | Matthews correlation: 57.06 -> 54.52 | Hurt |
| Use Ditto quantization and Quad GeLU approximation instead of the vanilla baseline | Bert-base · QQP | F1: 87.96 -> 87.72 | Hurt |
| Use Ditto quantization and Quad GeLU approximation instead of the vanilla baseline | Bert-base · QNLI | accuracy: 91.62 -> 91.78 | Helped |
| Use Ditto quantization and Quad GeLU approximation instead of the vanilla baseline | GPT2-base · Wikitext-103 | perplexity: 12.25 -> 13.78 | Hurt |
| Use Ditto secure quantization instead of PUMA | Bert-base · input length 128, LAN | runtime (s): 43.98 -> 30.58 | Helped |
| Use Ditto secure quantization instead of PUMA | Bert-base · input length 128, LAN | communication (GB): 10.59 -> 4.35 | Helped |
| Add Quad GeLU approximation to Ditto quantization (ablation) | Ditto on Bert-base · RTE | accuracy: 67.87 -> 67.82 | No clear effect |
| Add Quad GeLU approximation to Ditto quantization (ablation) | Ditto on Bert-base · CoLA | Matthews correlation: 54.17 -> 54.52 | Helped |
| Add Quad GeLU approximation to Ditto quantization (ablation) | Ditto on Bert-base · QQP | F1: 87.15 -> 87.72 | Helped |
| Add Quad GeLU approximation to Ditto quantization (ablation) | Ditto on Bert-base · QNLI | accuracy: 91.74 -> 91.78 | Helped |
| Add Quad GeLU approximation to Ditto quantization (ablation) | Ditto on Bert-base · inference | speedup over vanilla baseline: 1.41× -> 1.74× | Helped |
Surprising findings
- Quantization alone (without GeLU approximation) gives a 1.41~1.56x speedup, and GeLU Quad approximation raises it to 1.74~2.09x.
- MPCFormer incurs lower communication than Ditto on GPT2-base, attributed to the embedding layer's one-hot conversion done in MPC for Ditto.
Limitations
- GPT2 perplexity degrades noticeably (12.25 to 13.78 for GPT2-base), and CoLA utility drops on Bert.
- Evaluation is on a single CPU machine in LAN for main results; WAN results only in appendix.
- Upcast relies on a heuristic bias assumption that inputs lie in a bounded range.
Experimental setup
Datasets: GLUE RTE GLUE CoLA GLUE QQP GLUE QNLI Wikitext-103 BookCorpus English Wikipedia
Benchmarks: GLUE RTE GLUE CoLA GLUE QQP GLUE QNLI Wikitext-103
Metrics: accuracy Matthews correlation F1 perplexity runtime (s) communication (GB)
Baselines: MPCFormer PUMA SPU vanilla baseline Iron BumbleBee
Builds on: MPCFormer PUMA SecretFlow-SPU knowledge distillation 2-out-of-3 replicated secret sharing
Closest papers by meaning
- Nimbus: Secure and Efficient Two-Party Inference for Transformers (NeurIPS 2024 · similarity 0.87)
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · similarity 0.86)
- CipherPrune: Efficient and Scalable Private Transformer Inference (ICLR 2025 · similarity 0.85)
- An Efficient Private GPT Never Autoregressively Decodes (ICML 2025 · similarity 0.84)
- The Inhibitor: ReLU and Addition-Based Attention for Efficient Transformers (Student Abstract) (AAAI 2024 · similarity 0.84)
- Encryption-Friendly LLM Architecture (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.