Research map / Architectures and efficiency / RNNs and normalization

Transcoders find interpretable LLM feature circuits

NeurIPS 2024 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces transcoders (wide sparse ReLU MLPs approximating MLP sublayers) and a weights-based circuit analysis method whose attributions factorize into input-invariant and input-dependent terms, applied to GPT2-small and Pythia models.

Problem

Circuit analysis through MLP sublayers is hard because SAE features are dense combinations of many neurons with nonlinearities, and existing attribution methods (patching, input-times-gradient) only describe MLP behavior on a single input, not input-invariantly.

Approach

Replace each MLP sublayer with a wide sparse ReLU transcoder trained to approximate the MLP's output; then the attribution between transcoder features factorizes into the input-dependent activation times an input-invariant dot product of the earlier decoder vector with the later encoder vector, enabling weights-based circuit finding and de-embedding analysis.

Key results

• Transcoders match or outperform SAEs on sparsity-faithfulness Pareto frontiers across GPT2-small, Pythia-410M and Pythia-1.4B, with the gap widening on larger models.
• In a blind interpretability comparison of 50 transcoder vs 50 MLP-in SAE features in Pythia-410M layer 15, transcoders had 41 interpretable features vs 38 for SAEs.
• In the GPT2-small greater-than circuit, a handful of MLP10 transcoder features recover most of the model's performance, sparser than MLP10 neurons.

Measured effects reported in the paper

Surprising findings

• The 24th-highest-variance MLP10 transcoder feature had a flat direct logit attribution boosting all tokens equally and caused a sharp performance drop.

Limitations

• Transcoders are approximations and are approximately as unfaithful to model computation as SAEs by cross-entropy loss.
• Circuit analysis treats attention patterns (QK scores) as fixed and does not explain attention computation.
• Case studies are qualitative and limited in number.

Experimental setup

Datasets: OpenWebText

Benchmarks: OpenWebText (3.2M tokens) sparsity-faithfulness evaluation greater-than circuit (GPT2-small)

Metrics: L0 sparsity cross-entropy loss increase (faithfulness) human interpretability ratings mean probability difference

Baselines: sparse autoencoders on MLP outputs sparse autoencoders on MLP inputs MLP neurons

Builds on: sparse autoencoders attribution patching direct logit attribution greater-than circuit analysis

Closest papers by meaning

• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · similarity 0.91)
• Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition (ICLR 2025 · similarity 0.90)
• DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing (NeurIPS 2026 · similarity 0.89)
• The Computational Complexity of Circuit Discovery for Inner Interpretability (ICLR 2025 · similarity 0.89)
• Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis (ICML 2025 · similarity 0.88)
• Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference (AAAI 2025 · similarity 0.88)