Research map / Large language models / Instruction and fine-tuning
Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that L1 crosscoders produce spurious chat-only latents due to complete shrinkage and latent decoupling, and proposes Latent Scaling plus BatchTopK crosscoders to identify genuinely chat-specific latents in Gemma 2 2B base vs chat.
Problem
Crosscoder model diffing labels latents with near-zero base decoder norm as chat-only, but L1 training loss can misattribute shared concepts as chat-specific, so it is unclear which chat-only latents are genuinely chat-specific.
Approach
Identify two L1-induced failure modes (complete shrinkage and latent decoupling), measure them with Latent Scaling (least-squares scale of the chat decoder on base activations, split into reconstruction and error terms), and show BatchTopK crosscovers avoid these artifacts and yield more interpretable chat-specific latents.
Key results
- BatchTopK crosscoder top-50% Delta_norm latents give 50% KL reduction on first 9 tokens versus 6% for bottom 50%, while Delta_norm fails for L1 crosscoders.
- In the L1 crosscoder, most chat-only latents fail the chat-specificity test (nu_r < 0.5 and nu_eps < 0.2), whereas most BatchTopK chat-only latents pass.
- Chat template tokens are the last-token trigger for 40% of chat-only latents in the BatchTopK crosscoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BatchTopK sparsity instead of L1 sparsity | Crosscoder model diffing · Gemma-2-2b vs. Gemma-2-2b-it chat-only latents | Genuine chat-specific latent identification | Helped |
| Rank L1 crosscoder latents with Latent Scaling instead of Δ_norm | L1 crosscoder · 512 chat interactions, first 9 response tokens | KL divergence reduction from selected latents | Helped |
| Rank BatchTopK crosscoder latents by Δ_norm | BatchTopK crosscoder · 512 chat interactions, first 9 response tokens | KL divergence reduction: 50% vs 6% reduction | Helped |
| Use Δ_norm to rank L1 crosscoder latents | L1 crosscoder · 512 chat interactions, full responses and first 9 tokens | KL divergence improvement from selected latents | No clear effect |
| Replace all latents rather than use the base activation | L1 and BatchTopK crosscoders · 512 chat interactions, full responses | KL divergence reduction: 59% | Helped |
| Replace all latents rather than use the base activation | L1 and BatchTopK crosscoders · 512 chat interactions, first 9 response tokens | KL divergence reduction: 78% | Helped |
| Use the chat reconstruction error instead of the chat reconstruction | L1 and BatchTopK crosscoders · 512 chat interactions, full responses | KL divergence reduction | Helped |
| Use the chat reconstruction error instead of the chat reconstruction | L1 and BatchTopK crosscoders · 512 chat interactions, first 9 response tokens | KL divergence reduction | Hurt |
| Replace activations with chat activations only at template tokens | Gemma-2-2b base-to-chat activation approximation · 512 chat interactions, full responses | KL divergence: 0.239 | Helped |
| Replace activations with chat activations only at template tokens | Gemma-2-2b base-to-chat activation approximation · 512 chat interactions, first 9 response tokens | KL divergence: 0.507 | Helped |
| Use concatenated decoder norm for BatchTopK selection instead of summing decoder norms (ablation) | BatchTopK crosscoder · Gemma-2-2b vs. Gemma-2-2b-it chat-specific latent identification | Number of genuine chat-specific latents | No clear effect |
| Select BatchTopK latents independently for each model (ablation) | BatchTopK crosscoder · Gemma-2-2b vs. Gemma-2-2b-it chat-specific latent identification | Number of genuine chat-specific latents | No clear effect |
Surprising findings
- Full replacement with either crosscoder gives nearly identical KL reductions (59% overall, 78% on first 9 tokens), but error replacement is more than twice as bad on first 9 tokens for both.
- Delta_norm ranking is anti-correlated with causal importance for L1 crosscoders: high-Delta_norm latents perform nearly identically to or worse than low-Delta_norm latents.
Limitations
- Analysis limited to small models (Gemma 2 2B) and middle layer only.
- Cannot distinguish truly novel chat-tuning latents from existing latents whose activation shifted.
- Error terms of BatchTopK crosscoder still carry substantial chat behavior information.
Experimental setup
Datasets: web text mixture chat data LMSYS
Benchmarks: LMSYS chat prompts (512 interactions)
Metrics: KL divergence Latent Scaling ratios (nu_r, nu_eps) autointerpretability detection score
Baselines: L1 crosscoder norm-based relative decoder difference (Delta_norm)
Builds on: crosscoder BatchTopK SAE L1 sparse autoencoder
Closest papers by meaning
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.86)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.86)
- Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants (NeurIPS 2026 · similarity 0.86)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.86)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.86)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.