Research map / Federated learning and privacy / Privacy attacks and secure learning

MERGE: Fast Private Text Generation

AAAI 2024 · Federated learning and privacy › Privacy attacks and secure learning · approach: Encryption-based private learning · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MERGE, a two-party MPC private text generation framework that reuses output hidden states as next-token embeddings and merges Transformer linear ops with constant attention, achieving large speedups on GPT-2 and T5.

Problem

Private inference for Transformer language models in natural language generation is slow under secure multi-party computation, because existing approximations target NLU and ignore embedding lookup and token sampling costs in auto-regressive decoding; prior methods also suffer convergence failures o

Approach

Embedding resending feeds the output hidden state directly as the next input embedding, bypassing embedding table queries, combined with a merge module that replaces dynamic self-attention with constant attention and fuses linear layers into precomputed matrices, trained with cosine alignment, embedding augmentation, and weighted distillation.

Key results

• MERGE achieves 7.75x total speedup over CrypTen on GPT-2 and 10.89x on T5 at sequence length 128.
• MERGE reduces communication cost by about 62% on GPT-2 at length 128 and reports ~80% reduction at length 512.
• Baselines MPCformer (sf2quad) and THE-X give out-of-time (OOT) results on all generation tasks, while MERGE keeps BERTscore within about 0.02 of the best ReLU-approximated MPCformer.

Measured effects reported in the paper

Surprising findings

• Embedding resending alone without training yields severely degraded generation quality (e.g., BERTscore 0.686 vs 0.924 on MultiWOZ), requiring alignment and augmentation training.
• Vanilla MPCformer quadratic-softmax and THE-X variants fail to converge for NLG, producing out-of-time results.

Limitations

• Scalability to larger LLMs is only estimated analytically, since CrypTen lacked multi-GPU support; experiments are limited to models up to 391M parameters.
• Speedup gains are reported under a simulated two-GPU client-server setup over 10 GbE, not a real deployment.

Experimental setup

Datasets: MultiWOZ 2.1 DailyDialog CommonGen E2E

Benchmarks: MultiWOZ 2.1 NLG CommonGen DailyDialog

Metrics: BERTscore BARTscore NIST Rouge-L METEOR CHRF++ inference time communication cost

Baselines: CrypTen MPCformer (sm2relu) MPCformer (sm2quad) THE-X

Builds on: MPCformer THE-X CrypTen

Closest papers by meaning

• Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.88)
• Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs (ICML 2025 · similarity 0.87)
• SentinelLMs: Encrypted Input Adaptation and Fine-Tuning of Language Models for Private and Secure Inference (AAAI 2024 · similarity 0.87)
• SynBench: A Benchmark for Differentially Private Text Generation (NeurIPS 2026 · similarity 0.86)
• InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy (NeurIPS 2025 · similarity 0.86)
• PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.86)