Research map / Large language models / Language, safety and interpretability
Concept-Guided Interpretability via Neural Chunking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Reflection Hypothesis and three chunking methods (discrete sequence chunking, population averaging, unsupervised chunk discovery) to extract concept-encoding population-activity chunks from RNNs and LLMs, with causal grafting/freezing interventions.
Problem
Neural network internal activity is hard to interpret; it is unclear whether population activity reflects recurring structure in training data and whether those structures can be extracted as interpretable units.
Approach
Treat recurring regions of population activity as cognitive-style "chunks" (balls around prototype activation vectors). Three extraction methods cover small-d RNNs (discrete clustering plus BPE-style merging), large-d LLMs with known labels (population averaging with a deviation threshold), and unlabeled settings (dictionary learning with cosine similarity). Chunks are validated by grafting them i
Key results
- PA-extracted chunks from LLaMA-3 predict the presence of words like 'cheese', 'cake', and 'cheesecake' across layers with low false positives, and outperform individual SAE concept neurons.
- Grafting early-layer (1-9) chunks raised concept occurrence on TREC prompts, e.g. ABBR from 14.9% to 55.9% and LOC from 10.7% to 47.5%.
- Grafting 'cheese' chunks raised its occurrence in generations from 0% to 90% (N=100) on the prompt 'Hi, how are you doing?'.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| graft the cake chunk | LLaMA-3 · Hi, how are you doing? (N=100) | cake occurrence: 1% -> 83% | Helped |
| graft the cheese chunk | LLaMA-3 · Hi, how are you doing? (N=100) | cheese occurrence: 0% -> 90% | Helped |
| graft the cheesecake chunk | LLaMA-3 · Hi, how are you doing? (N=100) | cheesecake occurrence: 0% -> 16% | Helped |
| graft concept chunks at layers 1–9 rather than not grafting | LLaMA-3 · TREC ABBR prompts | concept occurrence: 14.9% -> 55.9% | Helped |
| graft concept chunks at layers 1–9 rather than not grafting | LLaMA-3 · TREC DESC prompts | concept occurrence: 15.6% -> 49.0% | Helped |
| graft concept chunks at layers 1–9 rather than not grafting | LLaMA-3 · TREC ENTY prompts | concept occurrence: 12.6% -> 48.1% | Helped |
| graft concept chunks at layers 1–9 rather than not grafting | LLaMA-3 · TREC HUM prompts | concept occurrence: 11.9% -> 46.7% | Helped |
| graft concept chunks at layers 1–9 rather than not grafting | LLaMA-3 · TREC LOC prompts | concept occurrence: 10.7% -> 47.5% | Helped |
| graft concept chunks at layers 1–9 rather than not grafting | LLaMA-3 · TREC NUM prompts | concept occurrence: 11.5% -> 45.3% | Helped |
| graft at early layers (1–9) rather than middle or late layers | LLaMA-3 · TREC categories | concept occurrence | Helped |
| freeze the chunk associated with a target concept | LLaMA-3 · concept-targeted generation | target concept generation | Hurt |
| graft neural population chunks instead of grafting input characters | RNN · compositional sequence transfer | transfer accuracy | Helped |
Surprising findings
- Input grafting alone was ineffective at inducing compositional reuse in RNNs, whereas grafting neural chunks was effective.
- Later-layer grafting was less effective than early-layer grafting for concept occurrence.
Limitations
- Method is evaluated mainly on word-level and narrative concepts; generalization to other concepts is not fully established.
- Learned vocabularies can be larger than ground-truth vocabularies, possibly an artifact of clustering choices.
- Population averaging is fragile for prepositions and context-dependent words, which have diverse neural states.
Experimental setup
Datasets: synthetic character sequences ROCStories TREC Question Classification Emma (Project Gutenberg)
Benchmarks: ROCStories TREC Question Classification
Metrics: true positive rate false positive rate decoding accuracy concept occurrence rate under grafting chunk dictionary size
Baselines: pretrained sparse autoencoder on LLaMA-3 representation engineering input grafting
Builds on: sparse autoencoders representation engineering byte-pair encoding
Closest papers by meaning
- Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment (AAAI 2025 · similarity 0.86)
- Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences (ICLR 2025 · similarity 0.86)
- Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants (NeurIPS 2026 · similarity 0.86)
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · similarity 0.86)
- Analyze Feature Flow to Enhance Interpretation and Steering in Language Models (ICML 2025 · similarity 0.86)
- Restyling Unsupervised Concept Based Interpretable Networks with Generative Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.