Research map / Architectures and efficiency / Attention and transformers
IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IceFormer, a retraining-free method that accelerates self-attention on CPUs by casting top-k attention selection as k-NN search over transformed keys using a ranking-based Prioritized DCI index with incremental updates for causal masks.
Problem
Quadratic time and space cost of self-attention makes long-sequence Transformers and LLMs slow to run on CPU-only hardware, and existing efficient attention methods either need retraining or lose accuracy.
Approach
Embed keys and queries into a (d+1)-dimensional space so that argmax attention weight equals nearest-neighbour search without key-norm constraints, then retrieve top-k keys with a ranking-based k-NNS (Prioritized DCI) that supports incremental insertion for causal masking; only the retrieved keys enter the softmax.
Key results
- On LRA, IceFormer achieves on average a 7.63x attention-module speedup over the vanilla Transformer while retaining 98.6% of its accuracy.
- On ZeroSCROLLS with Vicuna-7b-v1.5-16k, IceFormer attains 99.6% of vanilla average score (23.8 vs 23.9) at 2.7x average attention speedup.
- IceFormer has the lowest peak memory among compared methods on all LRA tasks, saving up to 0.862 GB versus the vanilla Transformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla attention with IceFormer | Transformer · LRA ListOps | accuracy: 0.4255 -> 0.4153 | Hurt |
| Replace vanilla attention with IceFormer | Transformer · LRA Text | accuracy: 0.6019 -> 0.5978 | Hurt |
| Replace vanilla attention with IceFormer | Transformer · LRA Retrieval | accuracy: 0.6586 -> 0.6541 | Hurt |
| Replace vanilla attention with IceFormer | Transformer · LRA Image | accuracy: 0.4132 -> 0.4046 | Hurt |
| Replace vanilla attention with IceFormer | Transformer · LRA Pathfinder | accuracy: 0.7514 -> 0.7442 | Hurt |
| Replace vanilla attention with IceFormer | Transformer · LRA ListOps | attention time: 2.9208 -> 0.3766 | Helped |
| Replace vanilla attention with IceFormer | Transformer · LRA Text | attention time: 0.6933 -> 0.0921 | Helped |
| Replace vanilla attention with IceFormer | Transformer · LRA Retrieval | attention time: 8.3588 -> 0.8337 | Helped |
| Replace vanilla attention with IceFormer | Transformer · LRA Image | attention time: 4.9303 -> 0.5076 | Helped |
| Replace vanilla attention with IceFormer | Transformer · LRA Pathfinder | attention time: 0.9620 -> 0.3058 | Helped |
| Use IceFormer for prompt processing | Vicuna-7b-v1.5-16k · ZeroSCROLLS average | score: 23.9 -> 23.8 (99.6%) | Hurt |
| Use IceFormer for prompt processing | Vicuna-7b-v1.5-16k · ZeroSCROLLS average | attention time: 4.38 -> 1.60 (2.7×) | Helped |
Surprising findings
- Naively applying k-NNS to unnormalized keys fails to find the highest-attention keys, since key norms modulate attention weights.
- Ranking-based k-NNS (Prioritized DCI) was argued to suit attention better than bucketing-based LSH because attention depends on relative key rankings.
Limitations
- Accuracy retention depends on hyperparameter k, trading speed for accuracy; experiments applied IceFormer only to layers 16-31 of the LLM.
- Shared-QK Transformer training failed on Pathfinder, so some comparisons are missing; Longformer, Performer and Linformer were randomly initialised for extra parameters.
- Evaluated only on CPU-based attention-module timing and a single 7B LLM; speedup measured on the attention module rather than end-to-end inference.
Experimental setup
Datasets: Long Range Arena (ListOps, Text, Retrieval, Image, Pathfinder) Fashion-MNIST
Benchmarks: Long Range Arena ZeroSCROLLS LongEval ANN benchmarks (Fashion-MNIST k-NNS comparison)
Metrics: accuracy attention module wall-clock time speedup peak memory usage approximation error ZeroSCROLLS score recall latency
Baselines: Transformer Reformer LARA Nyströmformer H-Transformer-1D Longformer Performer Linformer H2O
Builds on: Prioritized DCI Reformer Transformer self-attention Locality Sensitive Hashing
Closest papers by meaning
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.89)
- Two Heads are Better than One: Simulating Large Transformers with Small Ones (NeurIPS 2025 · similarity 0.88)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.87)
- LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units (ICLR 2024 · similarity 0.87)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.86)
- Mini-Sequence Transformers: Optimizing Intermediate Memory for Long Sequences Training (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.