Research map / Architectures and efficiency / RNNs and normalization
Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ITDA, a training-cheap alternative to sparse autoencoders that builds a dictionary of prompt-token activations via greedy error-thresholded sampling and decodes with matching pursuit, and uses dictionary Jaccard overlap as a cross-model representation similarity measure.
Problem
SAEs are expensive to train (hundreds of millions of tokens, parameter counts exceeding the LLM) and their latents are not comparable across models, limiting interpretability and model comparison at scale.
Approach
Replace learned SAE encoders with a dictionary of actual training activations, built greedily by adding activations poorly reconstructed by the current dictionary, and decode test activations with matching pursuit; because atoms are tied to prompt-token pairs, dictionaries are comparable across models and their Jaccard overlap measures representation similarity.
Key results
- On the GPT-2 layer-matching task, ITDA Jaccard similarity reaches 0.88 (small) and 0.89 (medium), above relative representations (0.87, 0.78), linear CKA (0.69, 0.61) and SVCCA (0.50, 0.44).
- ITDAs train in about 8 minutes on GPT-2 small versus 2-8 hours per SAE, and need about 1% of SAE training tokens.
- ITDA reconstruction is generally better than ReLU SAEs on Pythia but worse than TopK SAEs, and substantially worse than SAEs on Gemma-2 2B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ITDA Jaccard similarity instead of the Relative representation method | GPT-2 Small layer matching · GPT-2 Small layer matching | accuracy: 0.87 -> 0.88 | Helped |
| Use ITDA Jaccard similarity instead of the Relative representation method | GPT-2 Medium layer matching · GPT-2 Medium layer matching | accuracy: 0.78 -> 0.89 | Helped |
| Use ITDA instead of CKA, SVCCA, or Linear Regression for layer matching | GPT-2 Small and GPT-2 Medium layer matching · GPT-2 Small and GPT-2 Medium layer matching | accuracy: GPT-2 Small: 0.16, 0.50, 0.69 -> 0.88; GPT-2 Medium: 0.07, 0.44, 0.61 -> 0.89 | Helped |
| Use ITDA rather than SAE training | Activation decomposition · GPT-2 Small training | training time: 100x faster | Helped |
| Use ITDA rather than SAE training | Activation decomposition · Activation decomposition | training tokens: 1% | Helped |
| Use ITDA instead of ReLU SAE | Pythia activation reconstruction · Pythia | cross entropy loss score | Helped |
| Use ITDA instead of TopK SAE | Pythia activation reconstruction · Pythia | cross entropy loss score | Hurt |
| Use ITDA instead of SAE | Gemma 2 2B activation reconstruction · Gemma 2 2B | cross entropy loss score | Hurt |
| Lower the reconstruction-loss threshold (ablation) | ITDA dictionary construction · Activation reconstruction | reconstruction loss | Helped |
Surprising findings
- ITDA reaches a higher layer-matching accuracy than other relative representation and absolute similarity measures despite using only a simple Jaccard index over prompt-token labels.
Limitations
- ITDA reconstruction and interpretability are worse than TopK and P-Annealing SAEs and substantially worse than SAEs on Gemma-2 2B.
- Interpretability evaluation on SAEBench is of questionable applicability to non-SAE methods, and large-model benchmarks were infeasible to run.
- Atom labels can be misleading, and negative activations are hard to interpret.
Experimental setup
Datasets: The Pile
Benchmarks: SAEBench layer-matching benchmark (Mistral GPT-2 instances)
Metrics: CE loss score layer matching accuracy Jaccard similarity
Baselines: ReLU SAE TopK SAE JumpReLU SAE P-Annealing SAE Linear CKA SVCCA Relative representation Linear regression
Builds on: relative representations sparse autoencoders matching pursuit SAEBench TopK SAEs
Closest papers by meaning
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.89)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.88)
- Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.88)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.87)
- Low-Rank Adapting Models for Sparse Autoencoders (ICML 2025 · similarity 0.87)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.