Research map / Large language models / Language, safety and interpretability
Priors in time: Missing inductive biases for language model interpretability
Ekdeep Singh Lubana, Can Rager, Sai Sumedh R. Hindupur, Valérie Costa, Oam Patel, Sonia Krishna Murthy, Thomas Fel, Greta Tuckute, Daniel Wurgaft, Eric Bigelow, Demba E. Ba, Melanie Weber and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a temporal predictive component with attention over past context to the SAE | Temporal Feature Analysis vs BatchTopK SAE · garden path sentences | similarity of verb to subject phrase codes across ambiguous vs control | Improved |
| add predictive component instead of standard SAE codes | Temporal Feature Analysis · synthetic stories with event boundaries (50 stories) | within-event code similarity | Improved |
| use predictive component instead of SAE codes or novel component | Temporal Feature Analysis · in-context graph (cognitive map) task | counterfactual edits matching graph structure: 4/9 nodes yield expected counterfactual; no other representation does | Improved |
| add temporal inductive bias (TFA) instead of standard SAE | Gemma-2-2B layer 15 · reconstruction (NMSE, variance explained) | reconstruction error | No clear effect |
| use prediction-only module without novel component (ablation) | Temporal Feature Analysis · reconstruction on Gemma-2-2B | reconstruction quality | Hurt |
| train Temporal Feature Analyzer on a deeper layer (layer 26, ~81% depth) instead of middle layer (ablation) | Temporal Feature Analysis · Llama-3.1-8B | predictive component dominance of latent norm | Hurt |
| add Gaussian input noise (ablation) | Temporal Feature Analysis vs SAEs · story codes | explained variance of un-noised data | Improved |
Closest papers by meaning
- LLM Interpretability with Identifiable Temporal-Instantaneous Representation (NeurIPS 2025 · similarity 0.91)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.90)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.89)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.88)
- A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
