Research map / Architectures and efficiency / Attention and transformers
Free Energy Mixer
Jiecheng Lu, Shihao Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LSE mixing (L) to softmax attention with conv (ablation) | FEM-SM · MAD benchmark | average accuracy: 76.3 -> 78.8 | Improved |
| add temperature learning (T) on top of C,L (ablation) | FEM-SM · MAD benchmark | average accuracy: 78.8 -> 79.4 | Improved |
| add outer gate (G) on top of C,L,T (ablation) | FEM-SM · MAD benchmark | average accuracy: 79.4 -> 80.2 | Improved |
| add low-rank convolution (C) to softmax attention (ablation) | FEM-SM · MAD benchmark | average accuracy: 74.7 -> 76.3 | Improved |
| add all FEM modules to GLA (ablation) | GLA · MAD benchmark | average accuracy: 64.2 -> 74.9 | Improved |
| add FEM modules to Mamba | Mamba · MAD benchmark | average accuracy: 69.3 -> 73.0 | Improved |
| add FEM modules to AFT | AFT · MAD benchmark | average accuracy: 52.2 -> 69.9 | Improved |
| replace softmax attention with FEM-SM | Transformer 1.3B · FineWeb-Edu LM (1.3B) | average rank: 4.56 -> 2.06 | Improved |
| replace GLA with FEM-GLA | GLA 340M · FineWeb-Edu LM (340M) | average rank: 9.38 -> 6.56 | Improved |
| replace softmax attention with FEM-SM | DeiT-Tiny · ImageNet-1K | top-1 accuracy: 72.20 -> 76.70 | Improved |
| replace softmax attention with FEM-SM | DeiT-Small · ImageNet-1K | top-1 accuracy: 79.90 -> 80.45 | Improved |
| replace GLA with FEM-GLA | GLA · Solar | forecasting error: 0.204 -> 0.188 | Improved |
Closest papers by meaning
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.86)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.85)
- SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference (NeurIPS 2026 · similarity 0.84)
- Attention Drift: What Auto-Regressive Speculative Decoding Models Learn (NeurIPS 2026 · similarity 0.84)
- XTC: Head-Aware Sampling by Excluding Top Choices (NeurIPS 2026 · similarity 0.84)
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
