Research map / Architectures and efficiency / Attention and transformers
The geometry of hidden representations of large transformer models
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio ansuini, Alberto Cazzaniga
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use plateau-layer (low ID) representations instead of the last hidden layer for nearest-neighbor remote homology | ESM-2 · SCOPe remote homology | neighbor overlap with superfamily labels: 0.4 -> 0.8 | Improved |
| use intermediate layer at ID minimum instead of final layer | iGPT · ImageNet 300-class labels | overlap with ground truth labels: peak chi_gt 0.15 (S), 0.27 (M), 0.35 (L) | Improved |
| increase model size | iGPT · ImageNet | peak overlap with labels: 0.15 -> 0.35 | Improved |
| increase model size | ESM-2 · ProteinNet | peak intrinsic dimension: 20 -> 32 at peak; plateau stays 5-7 | Mixed |
| extract representations after attention maps or MLP instead of after first norm layer (ablation) | iGPT-L · ImageNet | ID and overlap profiles | No clear effect |
| vary neighborhood size k from 1 to 50 (ablation) | overlap analysis · ESM-2 650M and iGPT-L | overlap profile shape | No clear effect |
Closest papers by meaning
- How Transformers Learn Structured Data: Insights From Hierarchical Filtering (ICML 2025 · similarity 0.87)
- Effective Context in Transformers: An Analysis of Fragmentation and Tokenization (NeurIPS 2026 · similarity 0.87)
- Banyan: Improved Representation Learning with Explicit Structure (ICML 2025 · similarity 0.87)
- Constrained Belief Updates Explain Geometric Structures in Transformer Representations (ICML 2025 · similarity 0.87)
- Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.87)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
