Research map / Architectures and efficiency / RNNs and normalization
Do Transformer Interpretability Methods Transfer to RNNs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether three transformer interpretability methods (contrastive activation addition, tuned lens, and quirky-model latent knowledge elicitation) transfer to RNNs such as Mamba and RWKV, and proposes state steering using the RNN's compressed state.
Problem
Whether interpretability methods designed for transformer language models (steering, latent prediction elicitation, latent knowledge elicitation) still work on modern RNN architectures like Mamba and RWKV.
Approach
Applies contrastive activation addition, tuned lens, and quirky-model probing unchanged to Mamba and RWKV, and introduces state steering, which computes steering vectors from the RNN's recurrent state rather than the residual stream.
Key results
- Steering with CAA works on Mamba 2.8b and RWKV-v5 7b, with somewhat smaller effects than on BTLM 3b and Llama 2 7b for many behaviors, but more stable across layers for RWKV.
- Tuned lens on Mamba and RWKV achieves lower perplexity than the logit lens at each layer, with perplexity decreasing monotonically with depth.
- Probes trained on easy Alice contexts transfer to hard Bob contexts with about 0.76-0.78 AUROC on Mamba versus 0.69-0.75 on the transformer; combining activation and state steering is not additive.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply contrastive activation addition to RNN residual streams | Mamba 2.8b-slimpj and RWKV-v5 7b · Seven alignment-relevant behaviors | behavior probability change: 0.15; 0.2 | Mixed |
| Use activation steering on RNNs instead of comparable transformers | Mamba 2.8b-slimpj and RWKV-v5 7b · Sycophancy and refusal steering | behavior probability change | Hurt |
| Steer with both the RNN residual stream and internal state | Mamba and RWKV-v5 · Alignment-relevant behaviors | behavior probability change | Mixed |
| Add state steering to activation steering | Mamba and RWKV-v5 · Alignment-relevant behaviors | behavior probability change | No clear effect |
| Use the tuned lens instead of the logit lens | Mamba and RWKV-v4 · Pile validation set | perplexity | Helped |
| Increase tuned-lens layer depth | Mamba and RWKV-v4 · Pile validation set | perplexity | Helped |
| Use tied embedding and unembedding matrices | Mamba tuned lens · Early layers | perplexity | Hurt |
| Train probes on Alice easy contexts and transfer them to Bob hard contexts | Mamba 2.8b-slimpj · Quirky-model binary classification tasks | probe AUROC: 0.76; 0.78; 0.72; 0.76; 0.71; 0.71 | Helped |
| Train unsupervised probes on all context combinations instead of Alice easy contexts (ablation) | Mamba 2.8b-slimpj · Bob hard contexts | probe AUROC: 0.71 -> 0.54; 0.71 -> 0.69 | Hurt |
| Train probes on Alice easy contexts and transfer them to Bob hard contexts | BTLM-3b-8k · Quirky-model binary classification tasks | probe AUROC: 0.69; 0.75; 0.66; 0.52; 0.70; 0.68 | Helped |
| Train unsupervised probes on all context combinations instead of Alice easy contexts (ablation) | BTLM-3b-8k · Bob hard contexts | probe AUROC: 0.70 -> 0.54; 0.68 -> 0.66 | Hurt |
Surprising findings
- Combining residual-stream and state steering is not additive: the joint effect is smaller than the sum of individual effects.
- Probes on Mamba transferred from easy to hard contexts slightly better than on the transformer BTLM-3b-8k.
Limitations
- Only mechanistic or circuit-based interpretability tools were not explored.
- Steering effects were smaller on RNNs for some behaviors such as sycophancy and refusal.
- Only a few RNN and transformer models at limited scales were evaluated.
Experimental setup
Datasets: OpenHermes 2.5 The Pile Rimsky et al. (2023) behavior dataset Mallen and Belrose quirky datasets
Benchmarks: Rimsky et al. behavior steering dataset Pile validation (perplexity) quirky model probing (Alice/Bob)
Metrics: change in behavior probability under steering perplexity / bits per byte AUROC
Baselines: Llama-2-7b-chat BTLM-3b-8k Logit lens BTLM probes
Builds on: Contrastive activation addition Tuned lens Quirky models (Mallen and Belrose 2023) LoRA
Closest papers by meaning
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.87)
- Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures (ICLR 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.86)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.86)
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.86)
- Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.