Research map / Large language models / Language, safety and interpretability

Language Model Inversion

ICLR 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that next-token probability vectors of language models leak enough information to invert and recover hidden prompts, using a T5 encoder-decoder conditioned on unrolled log-probabilities plus a binary-search logit extraction method for API access.

Problem

Recovering a hidden prompt from a language model's next-token probability distribution (language model inversion), in settings where providers expose only partial probabilities, top-K, or text output.

Approach

Unroll the log-probability vector over the vocabulary into a fixed-length sequence of pseudo-embeddings that a pretrained T5 encoder consumes via cross-attention, training a decoder to reconstruct the prompt; additionally extract full logits through an API using logit-bias binary search.

Key results

• On Llama-2 7B, the inverter reaches BLEU 59.21 and token F1 77.8 on Instructions-2M, with 26.6% exact match, versus 36.5% exact match for the best (oracle) jailbreak baseline.
• On Llama-2 7B chat, the inverter achieves BLEU 58.26 and exact match 23.4%, while all jailbreak strings fail to exceed 5.4% exact match on average.
• Inverters trained on 7B transfer poorly to 13B and 70B models, with token F1 dropping from about 77 to about 53-55 on Instructions-2M.

Measured effects reported in the paper

Surprising findings

• Next-token probabilities retain prompt information even when the KL divergence between distributions for a synonym-swapped prompt has decayed, since low-probability tokens carry residual signal.
• The inverter outperforms sample-based inversion from argmax text outputs by a large margin in BLEU on the chat model, indicating logit vectors carry more usable information than sampled text.

Limitations

• Inverter must be retrained for each target model; transfer to 13B/70B models degrades substantially.
• Exact recovery is limited (about 27% on the base model); proper nouns are often mis-recovered.
• Evaluated only with T5-base (222M) inverters; the limits of larger inverters are not explored.

Experimental setup

Datasets: Instructions-2M Alpaca Anthropic HH Supernatural Instructions Self-Instruct Dolly ShareGPT Unnatural Instructions ChatBot Arena WizardLM GPTeacher T0

Benchmarks: Instructions-2M Alpaca Code Generation Anthropic HH

Metrics: BLEU token F1 exact match cosine similarity of text embeddings

Baselines: Jailbreak strings GPT-4 few-shot GPT-3.5 few-shot Sample inverter

Builds on: T5 Llama-2

Closest papers by meaning

• Better Language Model Inversion by Compactly Representing Next-Token Distributions (NeurIPS 2025 · similarity 0.93)
• Large Language Model Unlearning via Embedding-Corrupted Prompts (NeurIPS 2024 · similarity 0.86)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.86)
• From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.86)
• Physics of Language Models: Part 3.2, Knowledge Manipulation (ICLR 2025 · similarity 0.86)
• Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.86)