Research map / Adversarial robustness and security / Black-box and transfer attacks
Stealing part of a production language model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that black-box production LLM APIs leak their final embedding projection layer via logit/logprob and logit-bias queries, recovering the hidden dimension and the full projection matrix (up to affine/orthogonal symmetries) of OpenAI ada and babbage for under $20.
Problem
How much information about a production language model's internals can an adversary extract through a black-box API that exposes logprobs and logit bias, specifically its embedding projection layer and hidden dimension.
Approach
A top-down attack: the final projection W (l x h) has low rank h, so SVD of many collected full logit vectors reveals the hidden dimension and the column space of W, recovering W up to an h x h linear transform. Full logit vectors are reconstructed from top-K logprobs by forcing different tokens into the top-K via logit bias, using a reference token or a linear-system formulation; logprob-free bin
Key results
- Recovers exact hidden dimensions of OpenAI ada (1024) and babbage (2048) for under $20 total, with weight-matrix RMS around 5e-4 to 7e-4 up to an h x h transform.
- On open models GPT-2 Small, GPT-2 XL, Pythia, and LLaMA, the dimension is recovered within 0-2 (GPT-2 Small reported 757 vs 768 due to effective rank).
- The strongest logprob-free attack recovers about 18 bits of logit precision at 3.7 queries per logit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply hidden-dimension extraction to black-box production models with logit-bias and logprob access | OpenAI production models · ada, babbage, babbage-002, GPT-3.5-turbo-instruct, GPT-3.5-turbo-1106 | hidden-dimension extraction success: exactly correct | Helped |
| Recover the projection matrix using the embedding-projection extraction attack | OpenAI ada and babbage · ada and babbage | weight-matrix RMS: 5 · 10^{-4}; 7 · 10^{-4} | Helped |
| Use logprob-5 instead of logprob-4 to recover five rather than four logits per query | Top-5 logit-bias logit recovery · Top-5 logit-bias API | bits of precision: 23.0 -> 11.5 | Hurt |
| Use logprob-5 instead of logprob-4 to recover five rather than four logits per query | Top-5 logit-bias logit recovery · Top-5 logit-bias API | queries per logit: 0.25 -> 0.64 | Hurt |
| Use logprob-free one-of-n attack instead of the top-5 logprob method | Logit recovery · Logit-free API | bits of precision: 18.0 | Helped |
| Use logprob-free one-of-n attack instead of the top-5 logprob method | Logit recovery · Logit-free API | queries per logit: 3.7 | Mixed |
| Recover the effective rank rather than nominal width | GPT-2 Small hidden-dimension extraction · GPT-2 Small | stolen hidden dimension: 768 -> 757 | Mixed |
Surprising findings
- GPT-2 Small's final hidden activations have only 757 effective directions despite 768 dimensions in fp32.
- Recovering W exactly is impossible for transformers with residual connections and normalization, since orthogonal or invertible transforms leave outputs unchanged.
Limitations
- Recovers only the final projection layer up to an unavoidable affine or orthogonal symmetry, not the full model.
- Depends on logit-bias and multi-logprob API features that providers have since removed or restricted.
- Numerical instability for some attack variants (e.g., logprob-5) and float precision limits.
Experimental setup
Metrics: RMS error of recovered projection matrix numerical rank / stolen hidden dimension bits of logit precision queries per logit USD cost
Baselines: logprob-1 binary search hyperrectangle one-of-n
Builds on: Carlini et al. ReLU model stealing
Closest papers by meaning
- Better Language Model Inversion by Compactly Representing Next-Token Distributions (NeurIPS 2025 · similarity 0.87)
- Scalable Extraction of Training Data from Aligned, Production Language Models (ICLR 2025 · similarity 0.86)
- Stealth edits to large language models (NeurIPS 2024 · similarity 0.86)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.84)
- Eliciting Secret Knowledge from Language Models (NeurIPS 2026 · similarity 0.84)
- Representation Noising: A Defence Mechanism Against Harmful Finetuning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.