Research map / Large language models / AI and society
Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes converting activations of a CNN's first layer into natural-language shape descriptions by fine-tuning open LLMs (Mistral-7B, Mixtral-8x7B) as neural interpreters on binned, char-encoded activations with QLoRA.
Problem
Neural network interpretability methods give local, low-level explanations requiring expert effort and do not scale to large models; the paper seeks a way to translate internal activations into human-readable language.
Approach
Treat activation-to-language as a translation task: char-encode binned first-layer CNN activations, prepend a task prompt, and fine-tune a pretrained LLM autoregressively to output descriptions of the shapes present, using a semi-supervised (unsupervised then supervised) paradigm.
Key results
- Mistral-7B reaches 80% accuracy on all shapes and F1 0.8193 on held-out synthetic data, versus Mixtral-8x7B at 80% accuracy and F1 0.7059.
- Post-fine-tuning, Mixtral-8x7B derived the number of shapes correctly in 18 of 20 manually evaluated MNIST images; Mistral-7B identified shapes correctly in 12 of 20 cases.
- Mixtral-8x7B's pre-fine-tuning output on the synthetic evaluation set never produced descriptions, while post-fine-tuning it always produced shape predictions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mistral7b rather than Mixtral8x7b as the interpreter | Neural interpreter · Synthetic held-out data | F1 Score: 0.7059 -> 0.8193 | Helped |
| Use Mistral7b rather than Mixtral8x7b as the interpreter | Neural interpreter · Synthetic held-out data | FPR: 0.1282 -> 0.2273 | Hurt |
| Use Mistral7b rather than Mixtral8x7b as the interpreter | Neural interpreter · Synthetic held-out data | FNR: 0.2727 -> 0.3143 | Hurt |
| Fine-tune Mixtral8x7b on activation-description examples (ablation) | Mixtral8x7b · Synthetic held-out data | Shape prediction | Helped |
| Fine-tune Mixtral8x7b on activation-description examples (ablation) | Mixtral8x7b · MNIST test images | Images with a derived shape count: 18 out of 20 | Helped |
| Fine-tune Mixtral8x7b on activation-description examples (ablation) | Mixtral8x7b · MNIST test images | Description includes shape count and types | Helped |
| Fine-tune Mistral7b on activation-description examples (ablation) | Mistral7b · MNIST test images | Correct shape identification cases: 12 out of 20 cases | Helped |
| Add the delimiter '### Output: Number of Shapes:' to the Mistral7b prompt (ablation) | Mistral7b · MNIST test images | Correct shape identification cases: 12 out of 20 cases | Helped |
| Fine-tune Mixtral8x7b on activation-description examples (ablation) | Mixtral8x7b · Curve-count queries on MNIST | Generalization to curve questions | Helped |
| Use Mixtral8x7b rather than Mistral7b as the interpreter | Neural interpreter · Synthetic held-out data | Average Edit Distance: 1.60 -> 3.55 | Hurt |
Surprising findings
- The interpreter, without seeing images, answered a curve-counting question by combining learned shape concepts with pretraining knowledge, suggesting generalization beyond training labels.
- Without a specific hint in the prompt, the interpreter's performance on curve queries degraded severely.
Limitations
- Evaluation limited to a single first CNN layer on 28x28 MNIST-like images and simple shapes.
- Structured, template-based descriptions lack linguistic diversity, limiting flexibility and generalization.
- Mistral-7B often continued generating activations or shapes after accurate predictions; MNIST human evaluation was on small samples (20 cases).
Experimental setup
Datasets: MNIST synthetic shape images (15,000)
Benchmarks: held-out synthetic shape dataset MNIST test subset (100 images)
Metrics: precision recall F1 IoU ROUGE BLEU FPR FNR average edit distance accuracy (all shapes)
Baselines: Mixtral-8x7B
Builds on: QLoRA DeepDream Radford et al. 2017 unsupervised sentiment neuron Mistral-7B Mixtral-8x7B
Closest papers by meaning
- Automated Natural Language Explanation of Deep Visual Neurons with Large Models (Student Abstract) (AAAI 2024 · similarity 0.88)
- Automatically Interpreting Millions of Features in Large Language Models (ICML 2025 · similarity 0.88)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.87)
- Measuring Per-Unit Interpretability at Scale Without Humans (NeurIPS 2024 · similarity 0.87)
- Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.86)
- Concept-Guided Interpretability via Neural Chunking (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.