Research map / Large language models / Instruction and fine-tuning
Encryption-Friendly LLM Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an HE-friendly BERT-style transformer that combines LoRA fine-tuning (to avoid large ciphertext-ciphertext matrix multiplications) with Gaussian kernel attention (replacing softmax), enabling encrypted fine-tuning and inference under CKKS with near-plaintext GLUE accuracy.
Problem
Applying homomorphic encryption to LLM fine-tuning and inference is computationally prohibitive, mainly due to large ciphertext-ciphertext matrix multiplications from full fine-tuning and the difficulty of evaluating softmax under HE.
Approach
Use LoRA so that weight updates are small (turning large CCMMs into cheap PCMMs plus small CCMMs), and replace softmax attention with a division-free, max-free Gaussian kernel approximated by a polynomial for x<=0; also a modified AdamW (AdamW-HE) that moves epsilon outside the square root to avoid inverse-sqrt difficulties.
Key results
- LoRA+GK under HE achieves 6.94x fine-tuning speedup and 2.33x inference speedup over Full+SM on a single GPU (423.55s vs 61.03s fine-tuning).
- Average GLUE score under HE is 0.6621 vs 0.7068 for plaintext Full+SM fine-tuning across six tasks.
- Gaussian kernel attention is 6.24x faster than HETAL-style softmax under HE (1.44s vs 8.99s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace full fine-tuning with LoRA fine-tuning (ablation) | BERT-style model under HE with Softmax · Fine-tuning, 2 layers | time (s): 423.55 -> 76.97 | Helped |
| Replace Softmax with Gaussian kernel attention (ablation) | BERT-style model under HE with LoRA · Fine-tuning, 2 layers | time (s): 76.97 -> 61.03 | Helped |
| Replace Softmax with Gaussian kernel attention (ablation) | BERT-style model under HE with LoRA · Inference, 2 layers | time (s): 42.44 -> 26.5 | Helped |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model under HE · Inference, 2 layers | time (s): 61.84 -> 26.5 | Helped |
| Replace ciphertext-ciphertext matrix multiplication with plaintext-ciphertext matrix multiplication | Homomorphic matrix multiplication · 128×768 by 768×768 | time (s): 1.259 -> 0.275 | Helped |
| Replace Softmax with Gaussian kernel attention | Homomorphic attention computation · 128×128 matrices | time (s): 8.99 -> 1.44 | Helped |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model · GLUE CoLA | Matthews correlation: 0.2688 -> 0.1575 | Hurt |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model · GLUE MRPC | F1: 0.8304 -> 0.8147 | Hurt |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model · GLUE RTE | accuracy: 0.5884 -> 0.5993 | Helped |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model · GLUE STS-B | Pearson: 0.8164 -> 0.7997 | Hurt |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model · GLUE SST-2 | accuracy: 0.8991 -> 0.8188 | Hurt |
| Use LoRA+GK instead of full fine-tuning with Softmax | BERT-style model · GLUE QNLI | accuracy: 0.8375 -> 0.7827 | Hurt |
Surprising findings
- LoRA with softmax alone yields a 5.5x fine-tuning speedup, with GK providing the remaining gain.
- Plaintext LoRA+GK drops to 0.6772 average versus 0.7068 for full fine-tuning with softmax, with CoLA falling most sharply.
Limitations
- Only a 2-layer BERT with LoRA rank 2 is evaluated, with a small token length of 128
- Semi-honest threat model; model weights are not protected cryptographically
- Accuracy drop relative to plaintext softmax baselines, notably on CoLA
Experimental setup
Datasets: GLUE CoLA MRPC RTE STS-B SST-2 QNLI
Benchmarks: GLUE
Metrics: Matthews correlation F1 accuracy Pearson correlation execution time speedup factor
Baselines: Full+SM LoRA+SM HETAL NEXUS Powerformer
Builds on: LoRA CKKS JKLS HETAL AdamW
Closest papers by meaning
- EncryptedLLM: Privacy-Preserving Large Language Model Inference via GPU-Accelerated Fully Homomorphic Encryption (ICML 2025 · similarity 0.91)
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · similarity 0.91)
- Privacy-Preserving Embedding via Look-up Table Evaluation with Fully Homomorphic Encryption (ICML 2024 · similarity 0.87)
- CipherPrune: Efficient and Scalable Private Transformer Inference (ICLR 2025 · similarity 0.86)
- Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.86)
- PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.