Research map / Architectures and efficiency / Attention and transformers
Attention Projection Mixing with Exogenous Anchors
Jonathan Su
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an exogenous anchor with elementwise mixing (E-ExoFormer) vs gated attention | Gated Attention · FineWeb-Edu validation | perplexity: 14.64 -> 14.13 | Improved |
| add dynamic mixing module to E-ExoFormer (ablation) | E-ExoFormer · 6 MC tasks avg / PPL | average accuracy: 49.85 -> 50.27; PPL 14.13 -> 14.09 | Improved |
| add RMSNorm on anchor sources (E-ExoFormer vs No Norm) (ablation) | E-ExoFormer · FineWeb-Edu validation | perplexity: 14.30 -> 14.13 | Improved |
| add RMSNorm on anchor sources (E-ExoFormer vs No Norm) (ablation) | E-ExoFormer · 6 MC tasks | average accuracy: 49.69 -> 49.85 | Improved |
| add decoupled external anchor (E-ExoFormer instead of E-NuResFormer internal anchor) | residual mixing · FineWeb-Edu validation | perplexity: 14.15 -> 14.13 | Improved |
| use scalar vs elementwise mixing in NuResFormer (ablation) | NuResFormer · 6 MC tasks | average accuracy: 49.83 (scalar, Q/K norms) vs 49.68 (elementwise); PPL 14.15 elementwise best | Mixed |
| add gating to ResFormer without anchor normalization (naive combination) | ResFormer · 6 MC tasks | average accuracy: 49.65 -> 49.09 | Hurt |
| anchor on H1 instead of H0 embeddings (dynamic) (ablation) | Dynamic E-ExoFormer · FineWeb-Edu validation | perplexity: 14.09 -> 14.02 | Improved |
| anchor on H2 instead of H0 (dynamic) (ablation) | Dynamic E-ExoFormer · FineWeb-Edu validation | perplexity: 14.09 -> 14.09 | No clear effect |
| anchor on H1 instead of H0 (static) (ablation) | E-ExoFormer · FineWeb-Edu validation | perplexity: 14.13 -> 14.07 | Improved |
| add QKV-Norm post-mixing (ablation) | E-ExoFormer · FineWeb-Edu validation | perplexity: 14.13 -> 14.08 | No clear effect |
| add QKVG-Norm post-mixing (ablation) | E-ExoFormer · FineWeb-Edu validation | perplexity: 14.13 -> 14.07 | No clear effect |
Closest papers by meaning
- DisSparse: Pipelined Top-$p$ Sparse Attention for Long-Context LLM Serving (NeurIPS 2026 · similarity 0.85)
- Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency (ICLR 2026 · similarity 0.85)
- Don't Lose Focus: Activation Steering via Key-Orthogonal Projections (NeurIPS 2026 · similarity 0.84)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.84)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.84)
- Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
