Research map / Learning theory and algorithms / Kernels and regression theory
Interpretability Illusions in the Generalization of Simplified Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that simplified proxies (SVD, clustering, one-hot attention) of small Transformers match the original models in-distribution but diverge out-of-distribution on Dyck and code-completion generalization splits.
Problem
Simplified model representations used for mechanistic interpretability are typically validated only in-distribution; it is unclear whether they remain faithful to the original model on out-of-distribution generalization splits.
Approach
Construct explicit simplified proxy models (SVD dimensionality reduction, k-means clustering of keys/queries, one-hot attention) and measure their prediction and attention agreement with the original model across systematic generalization splits, revealing consistent generalization gaps in both directions.
Key results
- On Dyck, a rank-4 SVD/clustering simplification keeps near 100% prediction similarity in-distribution but deviates considerably on unseen structures where the original model generalizes almost perfectly, so simplifications underestimate generalization.
- One-hot attention is faithful on structure splits but on depth generalization the one-hot model slightly outperforms the original model, overestimating generalization.
- On CodeSearchNet, prediction similarity is consistently higher on Java than on unseen languages (paired t-test p <= 0.005), with the largest gap on the induction-head-like repeated-word predictions (75% ID vs 61% OOD for one head).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reduce second-layer key and query embeddings with SVD or clustering (ablation) | Dyck Transformer · Dyck, unseen structures | prediction similarity to the original model | Hurt |
| Reduce second-layer key and query embeddings with SVD or clustering (ablation) | Dyck Transformer · Dyck, Seen struct | prediction similarity to the original model | No clear effect |
| Replace second-layer soft attention with one-hot attention (ablation) | Dyck Transformer · Dyck, unseen depth | accuracy | Helped |
| Replace second-layer soft attention with one-hot attention (ablation) | Dyck Transformer · Dyck generalization splits other than unseen depth | approximation quality | No clear effect |
| Reduce key and query embeddings using SVD (ablation) | CodeSearchNet Transformer · Unseen programming languages | prediction similarity to the original model | Hurt |
| Reduce key and query embeddings using SVD (ablation) | CodeSearchNet Transformer · Java, unseen depth | prediction similarity to the original model | No clear effect |
| Simplify the fourth head of the fourth layer with rank-16 SVD (ablation) | CodeSearchNet Transformer · Repeated word, Java versus unseen languages | prediction similarity to the original model: 75% on in-domain samples; 61% on samples from unseen languages | Hurt |
| Reduce key and query embeddings with SVD (ablation) | CodeSearchNet Transformer · Repeated word predictions versus whitespace and keywords | prediction similarity to the original model | Mixed |
Surprising findings
- Hard (one-hot) attention simplification outperforms the original model on deep-nesting depth generalization, so the simplified proxy overestimates generalization.
- Data-dependent simplifications (SVD, clustering) fitted on in-distribution data can underestimate the original model's systematic generalization.
Limitations
- Only small-scale models (two-layer Dyck model with hidden size 32; four-layer code model) and a limited set of tasks were studied.
- Only one component (e.g., a single attention head) was simplified at a time; simplifying whole circuits may cause larger shifts.
- Black-box analysis cannot predict where deeper depths are embedded without reverse-engineering lower layers.
Experimental setup
Datasets: Dyck-(20,10) synthetic sentences CodeSearchNet
Benchmarks: Dyck generalization splits (Seen struct, Unseen struct, Unseen depth) CodeSearchNet (Java, Java unseen depth, JavaScript, PHP, Go)
Metrics: Jensen-Shannon divergence of attention same prediction rate bracket-matching accuracy next-character accuracy
Baselines: original (unsimplified) Transformer in-distribution evaluation (Seen struct / Java)
Builds on: Transformer induction heads Dyck language expressivity work
Closest papers by meaning
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.87)
- What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.86)
- Position: Deep Learning is Not So Mysterious or Different (ICML 2025 · similarity 0.86)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.86)
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.85)
- Position: Understanding LLMs Requires More Than Statistical Generalization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.