Research map / Large language models / Instruction and fine-tuning
SentinelLMs: Encrypted Input Adaptation and Fine-Tuning of Language Models for Private and Secure Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts pre-trained BERT/RoBERTa/mBERT to passkey-encrypted tokens (Blake2 tokenizer encryption, glide-reflection embedding transforms, vocabulary shuffling), then fine-tunes on encrypted data, reporting near-parity with original models on GLUE, CoNLL2003 NER, and XNLI.
Problem
Text sent to cloud-hosted language models can be intercepted or logged by servers, but naive encryption of inputs makes pre-trained models unable to use the resulting out-of-vocabulary or information-destroying tokens.
Approach
A one-time post-hoc adaptation (no further pre-training) that keyed-encrypts the tokenizer vocabulary with Blake2, applies repeated glide-reflections to token embeddings (distance-preserving affine transforms), and shuffles vocabulary and embedding indices, followed by fine-tuning on encrypted training data.
Key results
- Encrypted BERT/RoBERTa variants on GLUE generally stay within about 2% of original models, e.g. ERoBERTa (nglide=10) SST-2 94.61 vs 94.8 original.
- On CoNLL2003 NER, ERoBERTa (passkey llm123, nglide=10) reaches F1 95.37 vs 95.66 for original RoBERTa.
- Token nearest-neighbor recovery accuracy is about 0.009%, and ranking match after fine-tuning is about 0.01%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt and fine-tune BERT/RoBERTa on passkey-encrypted inputs instead of using the original model | BERT and RoBERTa · GLUE tasks | task metrics: 55.24 vs 67.8 (CoLA); 56.34 vs 56.34 (WNLI) | Mixed |
| Adapt and fine-tune mBERT on passkey-encrypted inputs | mBERT · XNLI in-language and zero-shot | accuracy: 77.3 -> 78.7; 75.2 -> 76.8; 74.2 -> 73.9; 74.3 -> 75.5; 70.5 -> 71.1; 63.8 -> 65.3 | Helped |
| Adapt and fine-tune RoBERTa on encrypted inputs | RoBERTa · CoNLL2003 NER | F1: 95.66 -> 95.37 | Hurt |
| Use passkey nlp2023 instead of llm123 (ablation) | Encrypted BERT and RoBERTa · GLUE tasks | task metrics: CoLA: 55.3 vs 55.24 (EBERT); 59.05 vs 60.57 (ERoBERTa) | Mixed |
| Increase glide-reflection iterations from 3 to 10 (ablation) | Encrypted models · GLUE tasks | task metrics: CoLA: 55.3 -> 59.05 (llm123 RoBERTa); MRPC: 89.76 -> 92.04 (llm123 RoBERTa) | Mixed |
| Fine-tune the encrypted WNLI model with learning rate 1e-4 instead of the default 2e-5 (ablation) | Encrypted WNLI models · GLUE WNLI | accuracy | Helped |
| Fine-tune original and adapted models on 10 examples for one epoch before testing token-recovery rankings | BERT and RoBERTa · GLUE tasks | token-ranking match: 0.01% | Hurt |
Surprising findings
- Some encrypted models outperform originals on certain tasks, e.g. ERoBERTa on MRPC (92.04 vs 90.2 F1) and CoLA (59.05 vs 67.8 is lower, but EBERT on MRPC 89.76 vs 88.9 is higher).
Limitations
- Evaluated only on encoder models; generative models are incompatible because of shared tokenizers and vocabularies.
- Some tasks (e.g. CoLA, RTE with RoBERTa) show gaps larger than 2%, attributed to needing more training iterations (unvalidated).
- Security relies on heuristic arguments that glide reflections plus shuffling are hard to invert; no formal security proof.
Experimental setup
Datasets: GLUE CoNLL2003 XNLI
Benchmarks: GLUE CoNLL2003 XNLI
Metrics: Matthews correlation accuracy F1 Pearson correlation precision recall
Baselines: BERT original RoBERTa original mBERT original
Builds on: BERT RoBERTa Blake2
Closest papers by meaning
- Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.88)
- Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.87)
- MERGE: Fast Private Text Generation (AAAI 2024 · similarity 0.87)
- DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.87)
- PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.87)
- Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.