Research map / Architectures and efficiency / Attention and transformers
FiX: Introducing Fine-grained Forget Gate into Softmax Attention
Runzhong Li, Renjie Liu, Li Qing, Bo Tang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use element-wise FiX forget gate instead of RoPE | softmax attention transformer · Common sense zero-shot avg | accuracy: 54.82 -> 55.24 | Improved |
| use FiX instead of scalar-gate FoX | softmax attention transformer · Common sense zero-shot avg | accuracy: 54.42 -> 55.24 | Improved |
| combine FiX with RoPE | FiX · Common sense zero-shot avg | accuracy: 55.24 -> 54.85 | No clear effect |
| combine FiX with RoPE | FiX · Training loss on FineWeb-Edu | loss | Improved |
| use FiX instead of RoPE or FoX | softmax attention transformer · Length extrapolation (CodeParrot, PG-19, NarrativeQA, up to 16,384) | per-token loss | Improved |
| use FiX instead of RoPE | softmax attention transformer · Length extrapolation beyond 4096 training length | per-token loss | Improved |
| use FiX instead of baselines | softmax attention transformer · BABILong (1k-4k context) | accuracy | Improved |
| set RMSNorm epsilon to 1e-30 instead of 1e-6 (ablation) | FiX · FineWeb-Edu training loss | training loss | Improved |
| use forget gate embedding in the first layer instead of low-rank projection (ablation) | FiX · FineWeb-Edu training loss | training loss | Improved |
| use Mamba-style activation instead of sigmoid for fine-grained gate (ablation) | FiX · FineWeb-Edu training loss | training loss | Improved |
| use Mamba-style activation instead of sigmoid for scalar gate (ablation) | VO-FoX (scalar gate) · FineWeb-Edu training loss | training loss | Hurt |
Closest papers by meaning
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.90)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.88)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.88)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.87)
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.87)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
