Research map / Large language models / Retrieval and knowledge
Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
Zifan He, Rui Ma, Yizhou Sun, Jason Cong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| offload memory-processing steps (relevancy, retrieval) of sparse attention to FPGA | GPU-FPGA system · sparse attention decoding, up to 1M tokens | end-to-end speedup: 1.04-1.49x | Improved |
| offload BM25 and top-k of single-stage RAG to FPGA | GPU-FPGA system · DRAGIN/FLARE/FS-RAG | memory processing speedup: 5.1-6.6x over BM25S | Improved |
| offload retrieval of two-stage RAG to FPGA (reranker stays on GPU) | GPU-FPGA system · two-stage RAG | memory processing speedup: 1.1-2.1x | Mixed |
| offload MemAgent decoding to FPGA in prefill-decode disaggregation | GPU-FPGA system · MemAgent, batch size 1 | speedup: 1.8x | Improved |
| increase batch size from 1 to 32 for sparse attention and RAG (ablation) | GPU-FPGA system · LServe geomean speedup | speedup: 1.19x -> 1.83x | Improved |
| increase batch size from 1 to 32 for MemAgent (ablation) | GPU-FPGA system · MemAgent geomean speedup | speedup: 1.85x -> 0.13x | Hurt |
| increase batch size from 1 to 32 for Memory as Context (ablation) | GPU-FPGA system · HMT/Titans geomean speedup | speedup: 1.48x -> 1.15x | Hurt |
| sequence length beyond 1M tokens on FPGA (ablation) | GPU-FPGA system · LServe and DeepSeek Attention | speed | Hurt |
| use threshold instead of top-k selection in SeerAttention-R on FPGA (ablation) | GPU-FPGA system · SeerAttention-R kernel speedup | kernel speedup: 1.8-2.2x -> 2.6-4.9x | Improved |
| run GPU-FPGA instead of GPU only for MemAgent energy | GPU-FPGA system · MemAgent | energy per request: 13662 -> 3202 J/req | Improved |
Closest papers by meaning
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (ICML 2024 · similarity 0.91)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.91)
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.90)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.90)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.90)
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
