Research map / Large language models / Language, safety and interpretability

Demystifying Embedding Spaces using Large Language Models

ICLR 2024 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Embedding Language Models (ELM), which use trained adapter layers to inject domain embeddings (movie and user embeddings from MovieLens) into a PaLM 2 LLM so it can describe and manipulate embeddings in natural language.

Problem

Embeddings are compact but hard to interpret directly; existing tools (t-SNE, UMAP, CAVs) are narrow or static, and text-only LLMs cannot describe arbitrary or hypothetical embedding points.

Approach

Train a two-layer MLP adapter that maps domain embedding vectors into the LLM's token-embedding space, using a two-stage schedule (adapter-only, then full fine-tuning) on 24 movie tasks and a user-profile task generated by PaLM 2-L, so the LLM treats embeddings as token-level inputs for interpretation, including interpolation and CAV-based extrapolation.

Key results

• Human raters (100 per task) and consistency metrics show ELM generalizes across most of the 24 movie tasks, with semantic consistency around 0.83-0.96 cosine on most single-movie tasks.
• On movie interpolation with Forrest Gump, ELM is significantly more semantically consistent than text-only PaLM 2-L and GPT-4 given ground-truth examples.
• On Amazon tasks, ELM reaches semantic cosine 0.89-0.96 and behavioral rank correlation 0.81-0.94 on most tasks.

Measured effects reported in the paper

Surprising findings

• User profiles show greater behavioral consistency as they are extrapolated further in CAV directions, attributed to profiles becoming more predictable.
• Generated Amazon item descriptions and endorsements were rated by humans as sometimes surpassing the quality of the ground-truth data.

Limitations

• Two-movie tasks show lower consistency, especially for similarities and differences.
• Consistency degrades for embeddings far from the data manifold; interpolation is linear while the true manifold may be nonlinear.
• Evaluation relies on PaLM 2-L-generated training targets and a DLM used as both re-embedder and ranker, and models are not released.

Experimental setup

Datasets: MovieLens 25M Amazon product dataset (Clothing, Shoes and Jewelry)

Benchmarks: MovieLens 25M movie tasks (24 tasks) user profile task Amazon tasks (appendix)

Metrics: semantic consistency (cosine) behavioral consistency (Spearman rank correlation, NDCG) human rating scores (0-1)

Baselines: PaLM 2-L GPT-4

Builds on: concept activation vectors Sentence-T5 PaLM 2 weighted alternating least squares matrix factorization prefix tuning

Closest papers by meaning

• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.86)
• EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.86)
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.86)
• Harnessing the Universal Geometry of Embeddings (NeurIPS 2025 · similarity 0.86)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.85)
• FACE: A General Framework for Mapping Collaborative Filtering Embeddings into LLM Tokens (NeurIPS 2025 · similarity 0.85)