Research map / Architectures and efficiency / Attention and transformers
Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts transformers (BERT-like LM and ViT/Swin) into fully polynomial form for CKKS-based homomorphic-encryption inference, using softmax-free scaled sigma-attention and a range-minimization training phase for LayerNorm and activations.
Problem
Transformers contain non-polynomial operations (softmax, LayerNorm, GELU) that cannot be evaluated under CKKS homomorphic encryption, and polynomial networks are unstable, so secure inference with transformers was not previously feasible.
Approach
Replace softmax with pointwise activation attention scaled by a length factor (sigma-scale attention), use BatchNorm in place of LayerNorm for vision (with extra stabilizers), and train with a range-minimization loss that narrows the inputs to non-polynomial layers (including LayerNorm variance) so low-degree polynomial approximations can be substituted.
Key results
- Polynomial 6-layer BERT-like model reaches perplexity 19.89 on WikiText-103 vs 18.98 for the vanilla transformer; 12-layer reaches 18.91 vs 16.89.
- Polynomial ViT on CIFAR-100 reaches 70.8% vs 73.4% for the original ViT; polynomial Swin on Tiny-ImageNet reaches 58.9% vs 59.4%.
- Secure inference over 128 tokens with the 6-layer 53.3M-parameter model takes 305 seconds under HElayers/HEaaN on A100 GPU, compared with 432 seconds for ResNet-152 over HE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Softmax attention with scaled-σ attention (ablation) | 6-layer BERT-like transformer · WikiText-103 | perplexity: 18.98 -> 19.07 | Hurt |
| Add range-minimization training (ablation) | P 6-layer BERT-like transformer · WikiText-103 | perplexity: 19.07 -> 19.15 | Hurt |
| Replace Softmax attention with scaled-σ attention (ablation) | 12-layer BERT-like transformer · WikiText-103 | perplexity: 16.89 -> 16.98 | Hurt |
| Add range-minimization training (ablation) | P 12-layer BERT-like transformer · WikiText-103 | perplexity: 16.98 -> 17.15 | Hurt |
| Convert the transformer to polynomial form | 6-layer BERT-like transformer · WikiText-103 | perplexity: 18.98 -> 19.89 | Hurt |
| Convert the transformer to polynomial form | 12-layer BERT-like transformer · WikiText-103 | perplexity: 16.89 -> 18.91 | Hurt |
| Convert ViT to HE-friendly polynomial form | ViT · CIFAR-100 | accuracy: 73.4 -> 70.8 | Hurt |
| Convert Swin to HE-friendly polynomial form | Swin Transformer · Tiny-ImageNet | accuracy: 59.4 -> 58.9 | Hurt |
| Add post-activation length scaling to GELU pointwise attention (ablation) | ViT · CIFAR-100 | accuracy: 68.41% to 70.87% (+2.46%) | Helped |
| Add post-activation length scaling to GELU pointwise attention (ablation) | ViT · CIFAR-10 | accuracy: 81.17% to 92.31% (+11.14%) | Helped |
| Use pointwise attention without length scaling (ablation) | NLP transformer · WikiText-103 | training stability | Hurt |
| Use pre- and post-scaling with S(L)=1/L (ablation) | NLP transformer with σ-attention · WikiText-103 | training convergence | Hurt |
Surprising findings
- Without length scaling, sigma-attention models' weights explode early in training, and pre+post scaling with 1/L fails to converge.
- Replacing LayerNorm with BatchNorm in sigma-attention transformers was highly unstable and failed to converge for NLP tasks.
Limitations
- Polynomial approximations add a 0.74 (6-layer) and 1.76 (12-layer) perplexity gap on WikiText-103 and remain a source of degradation.
- Softmax and LayerNorm are not directly approximated; instead, architecture changes and extra training are needed.
- Evaluated only on small-scale models and classification/language modeling; no secure training.
Experimental setup
Datasets: WikiText-103 CIFAR-100 CIFAR-10 Tiny-ImageNet
Benchmarks: WikiText-103 CIFAR-100 CIFAR-10 Tiny-ImageNet
Metrics: perplexity test accuracy latency
Baselines: vanilla softmax transformer ResNet-152 over HE
Builds on: CKKS Remez algorithm HElayers range loss for polynomial activations
Closest papers by meaning
- Encryption-Friendly LLM Architecture (ICLR 2025 · similarity 0.91)
- Nimbus: Secure and Efficient Two-Party Inference for Transformers (NeurIPS 2024 · similarity 0.89)
- DCT-CryptoNets: Scaling Private Inference in the Frequency Domain (ICLR 2025 · similarity 0.88)
- CipherPrune: Efficient and Scalable Private Transformer Inference (ICLR 2025 · similarity 0.88)
- The Inhibitor: ReLU and Addition-Based Attention for Efficient Transformers (Student Abstract) (AAAI 2024 · similarity 0.86)
- Ditto: Quantization-aware Secure Inference of Transformers upon MPC (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.