Research map / Large language models / Reasoning and chain of thought
SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SepLLM, a sparse attention architecture that keeps only initial, neighboring, and separator tokens' KV, applied training-free, from scratch, post-training, and in streaming, with FlexAttention-based kernels.
Problem
Quadratic attention cost and KV cache growth make LLM inference and training slow and memory-heavy for long inputs; prior training-free KV compression methods mismatch training, and StreamingLLM drops many intermediate tokens and degrades quality.
Approach
Observes that punctuation separator tokens receive disproportionate attention and hypothesizes they condense the information of the segment they end; restricts each token to attend only to initial tokens, all preceding separators, and a local window of n neighbors, both at inference and during training.
Key results
- Training-free SepLLM (n=256) on Llama-3-8B-Instruct reaches 77.18 flexible GSM8K-CoT vs 77.79 vanilla while using 47.36% of KV cache.
- From-scratch Pythia-160m SepLLM (n=64) cuts FLOPs to ~72% and reaches ~1.53x training speedup (1648 ms vs 2524 ms per iteration).
- On PG19 with 64K generated tokens on Llama-3-8B, SepLLM reaches perplexity 33.4 vs 37.9 for StreamingLLM at the same cache capacity c=800.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Retain separator KV rather than dropping it (ablation) | SepLLM versus StreamingLLM · GSM8K-CoT 8-shot | strict-match: 67.63 -> 77.18 | Helped |
| Retain separator KV rather than dropping it (ablation) | SepLLM versus StreamingLLM · MMLU 5-shot | overall accuracy: 62.10 -> 64.68 | Helped |
| Increase neighboring tokens from n=64 to n=128 (ablation) | SepLLM training-from-scratch · downstream tasks | performance | Helped |
| Make the first attention layer full attention (ablation) | SepLLM (n=64) · LAMBADA | perplexity: 40.08 -> 36.54 | Helped |
| Make the first and last attention layers full attention (ablation) | SepLLM (n=64) · LAMBADA | perplexity: 40.08 -> 33.41 | Helped |
| Increase separator cache capacity from s=32 to s=64 (ablation) | SepLLM · WikiText, 20K input | perplexity: 8.79 -> 8.72 | Helped |
| Increase streaming cache capacity and window size (ablation) | SepLLM · WikiText, long-input streaming | perplexity: 13.01 -> 12.09 | Helped |
| Use four common separators instead of the default nine (ablation) | SepLLM · GSM8K-CoT 8-shot | strict-match: 77.18 -> 76.85 | Hurt |
| Use only period and question mark as separators (ablation) | SepLLM · GSM8K-CoT 8-shot | strict-match: 77.18 -> 70.36 | Hurt |
| Replace separator tokens with fixed-interval tokens (ablation) | SepLLM · GSM8K-CoT 8-shot | strict-match: 77.18 -> 70.43 | Hurt |
| Remove initial tokens (ablation) | SepLLM streaming · WikiText, 5K input | perplexity: 13.1 -> 14.9 | Hurt |
| Remove positional-encoding shifting (ablation) | SepLLM streaming · WikiText, 5K input | perplexity: 13.1 -> 192.7 | Hurt |
Surprising findings
- Naive head-wise hybrid (sliding window on some heads) collapses MMLU to ~24-29% accuracy, far below SepLLM's 64.68%.
- Fixed-interval attention (FixLLM) performs notably worse than separator-based attention at similar KV usage.
Limitations
- Separator set is a hyperparameter chosen by hand; performance drops with fewer separator types.
- Training-from-scratch and post-training experiments use only small Pythia models; Llama-3 training details were unavailable so it was used only for training-free and streaming tests.
- Training-free gains rely on separator-attention patterns that may not hold across all architectures or languages.
Experimental setup
Datasets: Deduplicated Pile LongAlpaca WikiText PG19
Benchmarks: GSM8K-CoT MMLU ARC-Challenge ARC-Easy LAMBADA LogiQA PIQA SciQ PG19 WikiText Needle-in-a-Haystack
Metrics: accuracy flexible-extract strict-match perplexity runtime KV cache ratio FLOPs ratio wall-clock time
Baselines: StreamingLLM Vanilla full attention H2O SnapKV PyramidKV FixLLM
Builds on: StreamingLLM FlexAttention Pythia H2O SnapKV
Closest papers by meaning
- Efficient Streaming Language Models with Attention Sinks (ICLR 2024 · similarity 0.89)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.88)
- EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.88)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.88)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.88)
- UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.