Research map / Large language models / Language, safety and interpretability
Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRISM, a framework that clusters high-activation sentences of an LLM neuron or SAE feature and labels each cluster with an LLM, producing multiple concept descriptions per feature plus polysemanticity and CoSy-style description scores.
Problem
Automated neuron-level feature description methods in LLMs are not robust and assume monosemanticity, giving only a single description per feature despite widespread polysemanticity.
Approach
Sample high-percentile activations, embed them with a sentence encoder, cluster with k-means (k=5), and have an LLM label each cluster to yield multiple concept descriptions per feature; polysemanticity is scored by pairwise cosine similarity among these descriptions and description quality by adapted CoSy AUROC/MAD.
Key results
- On GPT-2 XL MLP neurons, PRISM (max) reaches AUROC 0.85 and 91.67% positive MAD, versus 0.64 and 65.00% for GPT-Explain.
- On Gemma Scope SAE features, Neuronpedia (AUROC 0.62, MAD 63.33%) slightly outperforms PRISM (max) (AUROC 0.54, MAD 38.33%).
- Random-sentence and random-description sanity checks drop PRISM (max) AUROC from 0.85 to 0.68 and 0.65 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the best-matching description from 5 PRISM descriptions rather than averaging descriptions | PRISM · GPT-2 XL MLP neurons | AUROC: 0.65 (0.61-0.69) -> 0.85 (0.78-0.91) | Helped |
| use the best-matching description from 5 PRISM descriptions rather than averaging descriptions | PRISM · Llama 3.1 8B Instruct MLP neurons | AUROC: 0.52 (0.48-0.55) -> 0.71 (0.63-0.78) | Helped |
| use 10 clusters instead of 5 (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.88 (0.83-0.93) | Helped |
| use 10 clusters instead of 5 (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (mean): 0.65 (0.61-0.69) -> 0.61 (0.58-0.64) | Hurt |
| use 1 cluster instead of 5 (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.75 (0.66-0.83) | Hurt |
| use 3 clusters instead of 5 (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.82 (0.74-0.89) | Hurt |
| replace Gemini 1.5 Pro with Qwen3 32B for description generation (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.85 (0.78-0.91) | No clear effect |
| replace Gemini 1.5 Pro with Phi-4 for description generation (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.82 (0.75-0.89) | Hurt |
| replace Gemini 1.5 Pro with DeepSeek R1 for description generation (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.79 (0.71-0.87) | Hurt |
| randomize sentence and highlight assignments across clusters (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.68 (0.59-0.76) | Hurt |
| randomly reassign descriptions across features (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.65 (0.56-0.74) | Hurt |
| sample from the 0.0 to 0.25 activation percentile interval instead of the top 1% (ablation) | PRISM on GPT-2 XL MLP neurons · GPT-2 XL MLP neurons | AUROC (max): 0.85 (0.78-0.91) -> 0.69 (0.61-0.77) | Hurt |
Surprising findings
- On Gemma Scope SAE features, PRISM (max) underperforms Neuronpedia and the PRISM (mean) scores are near chance in several settings.
- Evaluation ranking of methods is preserved but absolute scores drop substantially when non-Gemini LLMs are used as evaluators (e.g., PRISM max AUROC falls to 0.58 with Qwen3 32B).
Limitations
- Descriptions are limited to concepts expressible in natural language and may miss graph or algorithmic concepts.
- Fixed cluster count k=5 cannot guarantee capturing all salient concepts of a feature.
- Reliance on maximally activating corpus examples restricts coverage of rare or out-of-distribution patterns.
Experimental setup
Datasets: C4 English train split Cosmopedia
Benchmarks: CoSy-style AUROC/MAD on GPT-2 XL, Llama 3.1 8B Instruct, GPT-2 Small SAE, Gemma Scope FADE-based faithfulness (output-centric)
Metrics: AUROC MAD (percentage positive) polysemanticity score (pairwise cosine similarity) faithfulness
Baselines: MaxAct GPT-Explain Transluce-Explain Neuronpedia Output-Centric
Builds on: automated interpretability (Bills et al.) CoSy Output-Centric feature descriptions SASC
Closest papers by meaning
- Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness (ICLR 2025 · similarity 0.88)
- Measuring and Guiding Monosemanticity (NeurIPS 2025 · similarity 0.88)
- Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words (ICLR 2025 · similarity 0.88)
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · similarity 0.86)
- Concept-Guided Interpretability via Neural Chunking (NeurIPS 2025 · similarity 0.85)
- Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.