Research map / Architectures and efficiency / Attention and transformers
HyperMLP: An Integrated Perspective for Sequence Modeling
Jiecheng Lu, Shihao Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax with ReLU+L2Norm activation under feature-side upgrades (ablation) | attention (RoPE+conv) · MAD avg / NanoGPT loss | MAD avg: 80.15 -> 80.50; loss 3.0466 -> 3.0481 | No clear effect |
| add DPLR sequence mixing with lag layout in both R(1) and R(2) (ablation) | R-cg-q · MAD avg | MAD average: 66.78 -> 81.01 | Improved |
| add DPLR sequence mixing with lag layout (ablation) | R-cg-q · NanoGPT | final loss: 3.0828 -> 2.9956 | Improved |
| use reverse-offset (lag) layout for temporal mixing (ablation) | over-parameterized ReLU temporal mixing · MAD / NanoGPT | MAD avg and loss: 46.27 -> 81.82; loss 4.3567 -> 3.0497 | Improved |
| use lag layout (ablation) | over-parameterized GLU temporal mixing · MAD / NanoGPT | MAD avg and loss: 46.18 -> 78.52; loss 4.3662 -> 3.0386 | Improved |
| mix both R(1) and R(2) instead of R(1) only (ablation) | G-cg-q · MAD avg | MAD average: 77.77 -> 81.12 | Improved |
| replace ReLU with GLU activation (HyperGLU) (ablation) | HyperMLP matched budget · NanoGPT | final loss: 2.9956 -> 2.9865 | Improved |
| remove KV depthwise convolution (ablation) | HyperMLP / HyperGLU · MAD avg | MAD average: 81.01 -> 80.61; 81.12 -> 80.92 | No clear effect |
| compress VO side instead of QK side under fixed budget (ablation) | ReLU attention with conv · NanoGPT | final loss: R-c-q 3.1762 vs R-c-v 3.1323; S-c-q 3.1042 vs S-c-v 3.1384 | Mixed |
| add RoPE to softmax attention (ablation) | S · MAD avg | MAD average: 50.69 -> 72.38 | Improved |
| add HyperGLU in place of softmax attention | 1.3B / 100B tokens · 15-benchmark average rank | average rank: 4.22 -> 1.88 | Improved |
| add HyperMLP/HyperGLU in place of ReLU attention with KV conv | 340M / 15B tokens · 15-benchmark average rank | average rank: 5.13 -> 2.97 (HyperMLP); 2.31 (HyperGLU) | Improved |
Closest papers by meaning
- HyperAttention: Long-context Attention in Near-Linear Time (ICLR 2024 · similarity 0.87)
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.86)
- Fundamental limits of learning in sequence multi-index models and deep attention networks: high-dimensional asymptotics and sharp thresholds (ICML 2025 · similarity 0.85)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.85)
- LinearARD: Linear-Memory Attention Distillation for RoPE Restoration (NeurIPS 2026 · similarity 0.85)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
