Research map / Large language models / Language, safety and interpretability

Rep2Text: Decoding Full Text from a Single LLM Token Representation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Rep2Text trains an MLP adapter that maps a single last-token hidden state into a decoder LLM's embedding space so the decoder autoregressively reconstructs the original input text, recovering about half of 16-token Wikipedia sequences and outperforming adapted Vec2Text.

Problem

Whether and how much of the original input text can be recovered from a single last-token representation of an LLM, given that last-token states are optimized for next-token prediction.

Approach

A LLaVA-style inverter: a gated residual two-layer MLP adapter projects the target model's last-token hidden state into k pseudo-token embeddings of a decoder LLM, which then generates the input text; the decoder can be frozen (adapter-only) or LoRA-tuned jointly.

Key results

• On 16-token Wikipedia sequences, Mistral-7B-v0.1 representations decoded by Llama-3.1-8B reach 0.52 ROUGE-1, 0.32 ROUGE-2 and 0.81 BERTScore.
• Versus adapted Vec2Text at 32 tokens (Mistral-7B/Qwen-2.5-14B), Rep2Text improves Token F1 by 11% and entity score by 13% on Wikipedia.
• On OOD clinical data, Rep2Text reaches 0.26 Token F1 and 0.37 ROUGE-1 while Vec2Text scores no higher than 0.10 on nearly all metrics.

Surprising findings

• Larger decoder does not help: Qwen-2.5-32B slightly underperforms Qwen-2.5-14B.
• Sharing the target's model family with the decoder does not make inversion easier; Mistral-7B representations are most invertible.

Limitations

• Only models up to 32B evaluated.
• Trained on a sampled subset of The Pile due to limited compute.
• OOD clinical inversion alters critical details such as age and symptom duration.

Experimental setup

Datasets: The Pile (Wikipedia subset) Clinical patient notes (Hugging Face Sadaftb/clinical-nlp-patient-notes)

Benchmarks: Wikipedia 16-token test set (1,000 sequences) Clinical OOD summaries

Metrics: ROUGE-1 ROUGE-2 ROUGE-L Token F1 BLEU BERTScore structure score entity score topic score

Baselines: Vec2Text Vec2Text + Corrector

Builds on: LLaVA Vec2Text LoRA

Closest papers by meaning

• zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.89)
• Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.88)
• OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference (ICML 2025 · similarity 0.88)
• From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.88)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.87)
• Chunk-Distilled Language Modeling (ICLR 2025 · similarity 0.87)