Research map / Architectures and efficiency / Attention and transformers
The SuperActivator Mechanism: Transformers Concentrate Reliable Concept Signals in the Tail
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the SuperActivator mechanism, a theory and empirical study showing transformer attention concentrates reliable concept signals in a sparse high-activation tail, and uses tail tokens for concept detection and localization.
Problem
Concept activation scores from concept vectors are noisy and poorly separated between in-concept and out-of-concept tokens, making concept detection and localization unreliable.
Approach
Proves that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, with already-extreme activations growing fastest under positive skew, and exploits the resulting sparse high-activation tail (SuperActivators) via a top-quantile max-style detector and a local attribution target.
Key results
- SuperActivator detection achieves F1 of 0.56 on OpenSurfaces vs 0.49 for the strongest baseline (prompting), and 0.83 on COCO vs 0.69, with gains up to 0.14 F1.
- Detection typically peaks using only the top 5-10% of in-concept token activations.
- SuperActivator-derived attribution targets achieve higher attribution F1, insertion and lower deletion scores than global concept vector targets across nine attribution methods.
Surprising findings
- Most in-concept activations remain indistinguishable from out-of-concept activations even at the most separating layers (average proportion above the 98th percentile only ~0.4 at peak).
- Sparse Autoencoder concepts peak at higher sparsity thresholds than other concept types.
Limitations
- Theory covers a noiseless scalar reduction of a single concept-aligned head; the initial asymmetry that drives tail formation is left unexplained.
- Threshold-based detection requires validation data to set SuperActivator thresholds.
- Optimal sparsity varies across concepts and is not fully analyzed.
Experimental setup
Datasets: CLEVR COCO Broden-Pascal Broden-OpenSurfaces Sarcasm iSarcasm GoEmotions
Benchmarks: CLEVR COCO OpenSurfaces Pascal Sarcasm iSarcasm GoEmotions
Metrics: F1 insertion score deletion score attribution F1
Baselines: RandTok LastTok MeanTok CLS Prompt
Builds on: TCAV transformer circuits
Closest papers by meaning
- Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits (NeurIPS 2025 · similarity 0.85)
- Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.85)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.85)
- ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features (ICML 2025 · similarity 0.85)
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers (NeurIPS 2025 · similarity 0.85)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.