Research map / Architectures and efficiency / RNNs and normalization
Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
Usha Bhalla, Alex Oesterling, Claudio Mayrink Verdun, Himabindu Lakkaraju, Flavio Calmon
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add temporal contrastive loss to the SAE (ablation) | Temporal SAE (Pythia-160m) · MMLU sparse probing k=5 | semantics and context probe accuracy: removing it: semantics -0.07, context -0.1 | Improved |
| add temporal contrastive loss to the SAE (ablation) | Temporal SAE (Pythia-160m) · Pythia-160m smoothness (high) | smoothness (lower is smoother): removing it: +0.07 | Improved |
| add temporal loss to SAE training | Matryoshka SAE · Reconstruction (FVE, Pythia-160m) | FVE: 0.95 -> 0.94 | No clear effect |
| add temporal loss to SAE training | Matryoshka SAE · Autointerp (Pythia-160m) | autointerp score: 0.83 -> 0.81 | No clear effect |
| add temporal loss to SAE training | Matryoshka SAE · High-level feature smoothness (Gemma2-2b) | smoothness (lower is smoother): 0.15 -> 0.10 | Improved |
| contrast with a random earlier token instead of the previous token (ablation) | Temporal SAE (Pythia-160m) · MMLU sparse probing k=5 | context vs syntax probe accuracy: context +0.11, syntax -0.10, semantics -0.02 | Mixed |
| increase high-level split from 20% to 50% (ablation) | Temporal SAE (Pythia-160m) · MMLU sparse probing k=5 | semantics, context and syntax probe accuracy: semantics +0.02, context +0.09, syntax -0.08 | Mixed |
| decrease high-level split from 20% to 10% (ablation) | Temporal SAE (Pythia-160m) · Pythia-160m ablation | fraction alive, probes: fraction alive -0.07; semantics -0.01, context +0.01 | No clear effect |
| steer with T-SAE high-level features instead of baseline SAE features | Gemma2-2b steering · Steering success vs coherence (30 features) | steering Pareto frontier | Improved |
Closest papers by meaning
- Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures (NeurIPS 2026 · similarity 0.92)
- Automatically Interpreting Millions of Features in Large Language Models (ICML 2025 · similarity 0.91)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.91)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.91)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.91)
- LLM Interpretability with Identifiable Temporal-Instantaneous Representation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
