Research map / Federated learning and privacy / Privacy attacks and secure learning
Split-and-Denoise: Protect large language model inference with local differential privacy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Split-N-Denoise (SnD), which runs only the token embedding layer on the client, adds d_chi-privacy Laplacian noise to token embeddings, and denoises the server's returned embeddings locally with a transformer denoiser pre-trained on the server using public data and synthetic noise.
Problem
Protecting user text privacy during LLM inference (embedding-as-a-service) under local differential privacy, where the model is frozen and the server is untrusted, while keeping downstream utility of embeddings high.
Approach
Split the model so the client runs only the token embedding layer, perturb token embeddings with d_chi-privacy noise (norm-clipped), and let the client denoise the server's output embeddings using its raw token representations and known noise levels, a denoiser the server pre-trains on public data and shares with users.
Key results
- SnD improves AUC over baselines by about 22.2% (BERT, vs TokenEmbPriv), 7.3-12.3% (GPT-2), and over 10% on average (T5), per the paper.
- Average accuracy loss relative to no noise is 4.31% (BERT), 4.48% (T5), and 5.25% (GPT-2).
- SnD yields 5000x speedup over PCFT and 25000x over Iron in computation for one sample, and cuts user compute/memory by 95.3%/93.5% vs full inference on GPT2-XLarge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SnD with user-side denoising instead of the strongest listed baseline | BERT models · QQP | AUC: 22.2% | Helped |
| Use SnD with user-side denoising instead of the baselines | GPT models · MRPC | AUC: 7.3% to 12.3% | Helped |
| Use SnD with user-side denoising instead of the baselines | T5 models · RTE | AUC: over 10% | Helped |
| Replace user-side denoising with server-side denoising without noise-level knowledge (ablation) | SnD · BERT, MRPC | AUC: over 10% | Hurt |
| Remove norm clipping of privatized token embeddings (ablation) | SnD · Downstream tasks | accuracy | Hurt |
| Finetune the denoiser on 1000 samples rather than leaving it unchanged (ablation) | SnD with BERT Base fine-tuned on SST2 using 1,000 samples · MRPC, η=100 | AUC: 0.621 -> 0.642 | Helped |
| Increase BERT-base finetuning data without updating the denoiser (ablation) | SnD · MRPC, η=100 | AUC: 0.621 -> 0.575 | Hurt |
| Add privacy noise with SnD rather than use the no-noise setting | BERT Base · CoLA | accuracy: 0.751 -> 0.688 | Hurt |
Surprising findings
- Server-side denoising yields AUCs mostly below 0.55, while user-side denoising with noise-level knowledge performs much better.
- Larger models incur greater utility loss under the same privacy budget, since noise becomes harder to denoise after more layers.
Limitations
- Downstream accuracy on larger LLMs (LLaMA, OPT-6.7B) still needs improvement, per the authors.
- Denoiser size and user computation scale with the underlying LLM.
- Extension to sequence-to-sequence generation is open due to noise amplification in autoregressive decoding.
Experimental setup
Datasets: SQuAD AG News Financial Phrasebank Banking77 Health Fact Poem Sentiment TweetEval Hate Speech18 SMS Spam DailyDialog Yelp Review Full App Reviews
Benchmarks: CoLA QQP MRPC RTE
Metrics: accuracy AUC MSE cosine similarity mutual information attack accuracy
Baselines: TokEmbPriv Text2Text RAPT PCFT Iron server-side denoise
Builds on: d_chi-privacy split learning DP post-processing denoising Feyisetan et al. token embedding privatization RAPT
Closest papers by meaning
- Hidden No More: Attacking and Defending Private Third-Party LLM Inference (ICML 2025 · similarity 0.89)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.88)
- MERGE: Fast Private Text Generation (AAAI 2024 · similarity 0.88)
- Cape: Context-Aware Prompt Perturbation Mechanism with Differential Privacy (ICML 2025 · similarity 0.88)
- DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.87)
- Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.