Research map / Large language models / Language, safety and interpretability

Hidden No More: Attacking and Defending Private Third-Party LLM Inference

ICML 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that permutation- and noise-based private LLM inference schemes leak prompts via a vocabulary-matching hidden-state reversal attack, and proposes Cascade, a token-sharded multi-party inference scheme that resists it.

Problem

Private third-party LLM inference: hidden states sent to inference parties may reveal user prompts, and existing permutation/noise-based defenses were assumed secure without rigorous attack analysis.

Approach

A vocabulary-matching attack reverses hidden states token-by-token by running forward passes over the vocabulary (using the open weights and a proposal model with L1-threshold matching), which remains effective under sequence, hidden-dimension, and factorized permutations and noise. Defense: Cascade shards tokens across CompNodes and AttnNodes with (c, δ)-clustered-arithmetic sharding so that toke

Key results

• Vocab-matching attack perfectly decodes ~100% of 1000 held-out prompts (50 tokens) from Gemma-2-2B-IT and 99.7-100% from Llama-3.1-8B-Instruct across layers 1-26 without defenses.
• Under sequence-dim, hidden-dim, and factorized-2D permutations, the attack still decodes nearly all samples (e.g., 97.6-100%).
• Cascade with alpha=8 on BERT-Large runs in 1.58s vs 73.7s for Puma and 141.2s for MPCFormer, with 160x less communication than MPCFormer in the most expensive setting.

Measured effects reported in the paper

Surprising findings

• Permutation-based private inference schemes (PermLLM, STIP, Centaur) that were argued secure via large permutation spaces are in fact reversible in the open-weights setting.
• The attack works despite floating-point non-determinism by using L1-threshold fuzzy matching, with no meaningful loss in decoding accuracy.

Limitations

• Cascade does not provide cryptographic privacy guarantees; security is statistical and depends on sharding parameters (e.g., c, alpha >= 8 for good security).
• Layer 0 token embeddings are directly reversible, so Cascade is unsuitable when every individual token must be secret unless combined with SMPC.
• Attack evaluated mainly on 50-token prompts and two models; longer prompts only tested at small scale.

Experimental setup

Datasets: FineWeb-Edu FineWeb (CC-MAIN-2024-10 split)

Benchmarks: FineWeb-Edu

Metrics: percentage of perfectly decoded samples ROUGE-L BLEU runtime (seconds) communicated bytes (GB)

Baselines: Wan et al. 2024 MPCFormer Puma Vanilla inference PermLLM STIP Centaur

Builds on: Wan et al. 2024 hidden state inversion Morris et al. 2023 text embedding inversion PermLLM STIP Centaur MPCFormer Puma

Closest papers by meaning

• Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.89)
• Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.89)
• Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.89)
• DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.88)
• PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization (NeurIPS 2026 · similarity 0.88)
• TensorCommitments: A Lightweight Verifiable Inference for Large Language Models (NeurIPS 2026 · similarity 0.88)