Research map / Large language models / Language, safety and interpretability

Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Gaussian Concept Subspace (GCS), which fits a Gaussian over many logistic-regression probe vectors per concept and samples concept vectors from it, evaluated for faithfulness, plausibility, and emotion steering on Llama-2 and Gemma models.

Problem

Linear probing yields a single concept vector per concept that varies substantially with probing data and training, making concept vectors unstable for analysis and inference-time intervention.

Approach

Instead of one probe vector per concept, train many probes on random subsets of data to obtain observed vectors, fit a diagonal Gaussian over them, and use vectors sampled near the mean (within 1 sigma) as a robust concept subspace representation.

Key results

• Sampled vectors within 1 sigma show within-set cosine similarity above 0.93, versus roughly 0.8-0.9 for observed vectors.
• Sampled concept vectors at 1 sigma and 2 sigma achieve comparable or higher prediction accuracy than observed vectors across the three models.
• In emotion steering on Llama-2-7B-chat, GCS 1-sigma vectors reach joyfulness 2.98 (best, at strength 0.064) with coherence 4.86, versus mean difference's best joyfulness 1.80 with coherence 5.44.

Measured effects reported in the paper

Surprising findings

• Vectors sampled further from the mean (2 to 5 sigma) are generally less effective for steering than 1 sigma vectors.
• Mean difference steering produces less fluent text, including repetitive or unusual-unicode outputs.

Limitations

• Relies on GPT-4o-generated synthetic data, which may not reflect noisy real-world data.
• Plausibility evaluated only within a two-level concept hierarchy.
• Steering evaluation uses GPT-4o as judge, which may be inaccurate; distribution-level evaluation is simplified to 1,000 sampled vectors.

Experimental setup

Datasets: GPT-4o-generated concept descriptions (16 low-level concepts across sports event, populated place, animal, movie)

Benchmarks: GPT-4o-generated concept dataset (hold-out concept descriptions) emotion steering on joyful movie reviews

Metrics: cosine similarity (within observed, within sampled, observed classifier accuracy of concept vectors average cosine similarity between concepts joyfulness score coherence score

Baselines: Mean Difference One linear vector

Builds on: linear probing logistic regression concept vectors mean difference steering

Closest papers by meaning

• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.85)
• On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.85)
• Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models (NeurIPS 2025 · similarity 0.85)
• Learning Distribution-wise Control in Representation Space for Language Models (ICML 2025 · similarity 0.85)
• Provable In-Context Vector Arithmetic via Retrieving Task Concepts (ICML 2025 · similarity 0.85)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.85)