Research map / Architectures and efficiency / Attention and transformers
Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Understanding
Kai Liu, Zhan Su, Peijie Dong, Fengran Mo, Jianfei Gao, Shaoting Zhang, Kai Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace One-Round with End-to-End Multi-Round inference | SWA-3B-4k · LongBench | average score: 41.70 -> 50.99 | Improved |
| replace One-Round with End-to-End Multi-Round inference | RWKV-7-3B · LongBench | average score: 41.43 -> 48.03 | Improved |
| replace One-Round with End-to-End Multi-Round inference | SWA-3B-4k · NIAH 64k-256k | average accuracy: 3.40 -> 99.80 | Improved |
| preserve hidden states across rounds (EMR instead of NMR) (ablation) | SWA-3B-4k vs Qwen-2.5-3B · LongBench | average score: 48.37 -> 50.99 | Improved |
| use Non-End-to-End Multi-Round inference instead of One-Round | Qwen-2.5-3B · LongBench | average score: 47.38 -> 48.37 | Improved |
| use Non-End-to-End Multi-Round inference instead of One-Round | Qwen-2.5-7B · LongBench | average score: 54.60 -> 51.33 | Hurt |
| add early stopping (ablation) | SWA-3B-4k-EMR · NIAH | average accuracy: 99.87 -> 99.70 | No clear effect |
| chunk size 4096 instead of 512 with window 512 (ablation) | SWA EMR · NIAH | accuracy: 97.0 -> 0.0 | Hurt |
| scale to 7B parameters | SWA-7B-4k-EMR · LongBench | average score: 46.17 -> 53.86 | Improved |
| increase window size from 512 to 4096 at chunk 2048 (ablation) | SWA EMR · NIAH | accuracy: 0% -> 100% | Improved |
| increase chunk size from 512 to 4096 at window 4096 (ablation) | SWA EMR · NIAH | inference time: 646s -> 366s | Improved |
| use EMR instead of CompACT compression | SWA-3B-4k · HotpotQA | F1: 36.57 -> 54.25 | Improved |
Closest papers by meaning
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.91)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.90)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.90)
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.90)
- ReAttention: Training-Free Infinite Context with Finite Attention Scope (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
