Research map / Architectures and efficiency / Attention and transformers
Causal Attention with Lookahead Keys
Zhuoqing Song, peng sun, Huizhuo Yuan, Quanquan Gu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard causal attention with lookahead keys (CASTLE) | Transformer LM · FineWeb-Edu 50B tokens, 1.3B | validation loss: 2.426 -> 2.391 | Improved |
| replace standard causal attention with lookahead keys (CASTLE) | Transformer LM · FineWeb-Edu 50B tokens, 160M | validation loss: 2.798 -> 2.792 | Improved |
| add lookahead keys (CASTLE) | Transformer LM · 5-shot downstream average, 1.3B | average accuracy (%): 58.11 -> 60.50 | Improved |
| add sliding window to lookahead keys (CASTLE-SWL) | CASTLE · FineWeb-Edu 50B tokens | validation loss: 2.391 -> 2.389 (1.3B) | No clear effect |
| add causal keys alongside lookahead keys (ablation) | CASTLE · FineWeb-Edu 25B tokens, 120M | validation loss: 3.021 -> 2.920 | Improved |
| add lookahead keys while matching key count to baseline (ablation) | CASTLE-M-16 · FineWeb-Edu 25B tokens, 340M | validation loss: 2.742 -> 2.716 | Improved |
| apply SiLU to lookahead-key scores (ablation) | CASTLE-SWL-XL · FineWeb-Edu 25B tokens | training stability | Improved |
| keep SiLU instead of lowering learning rate to stabilize (ablation) | CASTLE-SWL-XL · FineWeb-Edu 25B tokens | validation loss: 2.520 -> 2.503 | Improved |
| use sliding window 512 instead of 64 for lookahead keys (ablation) | CASTLE-SWL · FineWeb-Edu 25B tokens, XL | validation loss: 2.525 -> 2.503 | Improved |
| vary sliding window between 128 and 1024 (ablation) | CASTLE-SWL · FineWeb-Edu 25B tokens, medium/XL | validation loss: medium: best-worst gap 0.0038; XL: 0.0099 | No clear effect |
| use SiLU instead of tanh clipping of lookahead scores (ablation) | CASTLE-M · FineWeb-Edu 50B tokens, 350M | validation loss: 2.634 -> 2.615 | Improved |
| use SiLU instead of constant scaling of lookahead scores (ablation) | CASTLE-SWL-XL · FineWeb-Edu 50B tokens | validation loss: 2.417 -> 2.389 | Improved |
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.85)
- Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling (ICML 2025 · similarity 0.85)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.85)
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.85)
- KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation (ICML 2024 · similarity 0.85)
- Learning to Focus: Causal Attention Distillation via Gradient‐Guided Token Pruning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
