Research map / Large language models / Language, safety and interpretability
Better Language Model Inversion by Compactly Representing Next-Token Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PILS, which losslessly compresses full next-token logprob vectors over multiple generation steps into D-dimensional representations via an alr transform, and trains a T5-base inverter to recover hidden prompts from them.
Problem
Recovering hidden prompts or system messages from language model outputs using next-token logprobs, which prior logprob-based method (L2T) only uses at a single generation step and which are expensive to obtain via APIs.
Approach
Next-token probability vectors lie in a D-dimensional subspace (softmax bottleneck), so an additive log-ratio transform plus selecting D+100 token indices losslessly compresses each step's logprobs to a D-dimensional vector; a T5 inverter then consumes sequences of these vectors across many generation steps, needing only about D+1 logprobs per step.
Key results
- PILS achieves 51% exact match recovery of hidden prompts for Llama 2 Chat on in-distribution data, versus 23% for the best previous method (L2T).
- On Alpaca Code, exact recovery of Llama 2 Chat prompts rises from 17% to 60%.
- An inverter trained on 16 steps gains 5-27 points of prompt recovery when evaluated with 32 steps at test time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PILS logprob sequences instead of L2T | Llama 2 Chat prompt inversion · 2M Instructions validation | exact match: 23% -> 51% | Helped |
| Use PILS logprob sequences instead of the previous best model | Llama 2 Chat prompt inversion · Alpaca Code | exact match: 17% -> 60% | Helped |
| Use 32 generation steps at test time instead of 16 | PILS trained on 16 steps · Test-time prompt recovery | prompt recovery: 5--27 points | Helped |
| Use PILS instead of Dory | Prompt inversion · Dory comparison | BLEU: 58--69 point improvements | Helped |
| Use PILS with multiple generation steps instead of one-step L2T | Prompt inversion · In-domain and out-of-distribution prompts | exact recovery rate: 2--3.5 times higher | Helped |
| Use PILS with one generation step rather than L2T (ablation) | Prompt inversion · Across metrics and datasets | inversion performance | No clear effect |
Surprising findings
- Verbatim repetition by the base model makes prompting baselines strong on OOD sets, where prompting beats PILS on the base model.
- Post-training (e.g., Llama 3 Instruct) makes inversion harder, with lower performance than Llama 2 Chat.
Limitations
- Storage cost of precomputed compressed logprobs becomes a scaling bottleneck (over 500 GB for 2M 16-step generations).
- Inversion performance drops substantially for post-trained, safety-tuned models such as Llama 3 Instruct.
Experimental setup
Datasets: 2M Instructions Alpaca Code Anthropic Helpful/Harmless (HH) Awesome GPT Prompts GPT Store
Benchmarks: 2M Instructions validation Alpaca Code Anthropic HH Awesome GPT Prompts GPT Store
Metrics: exact match BLEU token F1
Baselines: Output2Prompt (O2P) Logit2Text (L2T) L2T++ DORY
Builds on: Logit2Text (L2T) Output2Prompt (O2P) softmax bottleneck
Closest papers by meaning
- Language Model Inversion (ICLR 2024 · similarity 0.93)
- Stealing part of a production language model (ICML 2024 · similarity 0.87)
- Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks (AAAI 2025 · similarity 0.87)
- Hidden No More: Attacking and Defending Private Third-Party LLM Inference (ICML 2025 · similarity 0.87)
- SentinelLMs: Encrypted Input Adaptation and Fine-Tuning of Language Models for Private and Secure Inference (AAAI 2024 · similarity 0.86)
- SilentStriker: Toward Stealthy Bit-Flip Attacks on Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.