AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

FiX: Introducing Fine-grained Forget Gate into Softmax Attention

Runzhong Li, Renjie Liu, Li Qing, Bo Tang

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use element-wise FiX forget gate instead of RoPEsoftmax attention transformer · Common sense zero-shot avgaccuracy: 54.82 -> 55.24Improved
use FiX instead of scalar-gate FoXsoftmax attention transformer · Common sense zero-shot avgaccuracy: 54.42 -> 55.24Improved
combine FiX with RoPEFiX · Common sense zero-shot avgaccuracy: 55.24 -> 54.85No clear effect
combine FiX with RoPEFiX · Training loss on FineWeb-EdulossImproved
use FiX instead of RoPE or FoXsoftmax attention transformer · Length extrapolation (CodeParrot, PG-19, NarrativeQA, up to 16,384)per-token lossImproved
use FiX instead of RoPEsoftmax attention transformer · Length extrapolation beyond 4096 training lengthper-token lossImproved
use FiX instead of baselinessoftmax attention transformer · BABILong (1k-4k context)accuracyImproved
set RMSNorm epsilon to 1e-30 instead of 1e-6 (ablation)FiX · FineWeb-Edu training losstraining lossImproved
use forget gate embedding in the first layer instead of low-rank projection (ablation)FiX · FineWeb-Edu training losstraining lossImproved
use Mamba-style activation instead of sigmoid for fine-grained gate (ablation)FiX · FineWeb-Edu training losstraining lossImproved
use Mamba-style activation instead of sigmoid for scalar gate (ablation)VO-FoX (scalar gate) · FineWeb-Edu training losstraining lossHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy