AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

HyperMLP: An Integrated Perspective for Sequence Modeling

Jiecheng Lu, Shihao Yang

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace softmax with ReLU+L2Norm activation under feature-side upgrades (ablation)attention (RoPE+conv) · MAD avg / NanoGPT lossMAD avg: 80.15 -> 80.50; loss 3.0466 -> 3.0481No clear effect
add DPLR sequence mixing with lag layout in both R(1) and R(2) (ablation)R-cg-q · MAD avgMAD average: 66.78 -> 81.01Improved
add DPLR sequence mixing with lag layout (ablation)R-cg-q · NanoGPTfinal loss: 3.0828 -> 2.9956Improved
use reverse-offset (lag) layout for temporal mixing (ablation)over-parameterized ReLU temporal mixing · MAD / NanoGPTMAD avg and loss: 46.27 -> 81.82; loss 4.3567 -> 3.0497Improved
use lag layout (ablation)over-parameterized GLU temporal mixing · MAD / NanoGPTMAD avg and loss: 46.18 -> 78.52; loss 4.3662 -> 3.0386Improved
mix both R(1) and R(2) instead of R(1) only (ablation)G-cg-q · MAD avgMAD average: 77.77 -> 81.12Improved
replace ReLU with GLU activation (HyperGLU) (ablation)HyperMLP matched budget · NanoGPTfinal loss: 2.9956 -> 2.9865Improved
remove KV depthwise convolution (ablation)HyperMLP / HyperGLU · MAD avgMAD average: 81.01 -> 80.61; 81.12 -> 80.92No clear effect
compress VO side instead of QK side under fixed budget (ablation)ReLU attention with conv · NanoGPTfinal loss: R-c-q 3.1762 vs R-c-v 3.1323; S-c-q 3.1042 vs S-c-v 3.1384Mixed
add RoPE to softmax attention (ablation)S · MAD avgMAD average: 50.69 -> 72.38Improved
add HyperGLU in place of softmax attention1.3B / 100B tokens · 15-benchmark average rankaverage rank: 4.22 -> 1.88Improved
add HyperMLP/HyperGLU in place of ReLU attention with KV conv340M / 15B tokens · 15-benchmark average rankaverage rank: 5.13 -> 2.97 (HyperMLP); 2.31 (HyperGLU)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy