Research map / Architectures and efficiency / Attention and transformers
Prototype Transformer: Towards Language Model Architectures Interpretable by Design
Yordan Yordanov, Matteo Forasassi, Bayar Menzat, Ruizhi Wang, Chang Qi, Markus Kaltenberger, Amine M'Charrak, Tommaso Salvatori, Thomas Lukasiewicz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace self-attention with prototype mixer (ProtoT) | LLaMA-3 style transformer · FineWeb-Edu large-scale | test perplexity: 25.8 -> 29.5 | Hurt |
| use ProtoT instead of DeltaNet linear attention | linear-compute LM · FineWeb-Edu large-scale | test perplexity: 31.5 -> 29.5 | Improved |
| increase context length from 256 to 2048 with fixed model size (ablation) | ProtoT · FineWeb-Edu default size | test perplexity: 80.7 (1024) -> 83.0 (2048) | Mixed |
| increase hidden size to 512 (ablation) | ProtoT · FineWeb-Edu ctx=2048 | test perplexity: 83.0 -> 70.2 | Improved |
| increase layers to 12 (ablation) | ProtoT · FineWeb-Edu ctx=2048 | test perplexity: 83.0 -> 71.8 | Improved |
| increase prototypes from 32 to 64 (ablation) | ProtoT · FineWeb-Edu ctx=2048 | test perplexity: 83.0 -> 76.1 | Improved |
| use ProtoT instead of LLaMA self-attention for generation | large-scale LM · LLM-judge text generation | Elo: 975.18 -> 1021.24 | Improved |
| use ProtoT instead of LLaMA for GLUE fine-tuning | large-scale LM · GLUE average | GLUE score: 71.6 -> 67.6 | Hurt |
| use prototype write gate features instead of LLaMA SAE features | interpretability analysis · LLM-judged disentanglement | disentanglement score: 5.91 -> 6.52 | Improved |
| disrupt female prototype L9 P7 by re-initialization (ablation) | ProtoT · women word probability | delta probability: -16.60% | Hurt |
| disrupt control prototype L9 P2 (ablation) | ProtoT · women word probability | delta probability | No clear effect |
| prototype interventions across concepts (ablation) | ProtoT · FineWeb validation | perplexity change: Max +5.58% (COVID) | No clear effect |
Closest papers by meaning
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.87)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.87)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.86)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.86)
- Latent Principle Discovery for Language Model Self-Improvement (NeurIPS 2025 · similarity 0.86)
- Language Modeling by Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
