Research map / Architectures and efficiency / Attention and transformers
Q-Delta: Beyond Key–Value Associative State Evolution
Sumin Park, Seojin Kim, Noseong Park
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add query-conditioned feedback to the delta-rule state update (Q-Delta vs Gated DeltaNet) | Gated DeltaNet · Zero-shot commonsense avg, 340M | average accuracy: 46.01 -> 47.24 | Improved |
| add query-conditioned feedback (Q-Delta vs Gated DeltaNet), 1.3B | Gated DeltaNet · Zero-shot commonsense avg, 1.3B | average accuracy: 52.77 -> 53.47 | Improved |
| add query feedback for long-context retrieval | Gated DeltaNet · S-NIAH average, 1.3B | accuracy: 83.51 -> 90.02 | Improved |
| add query feedback for LAMBADA perplexity at 340M | Gated DeltaNet · LAMBADA, 340M | perplexity: 36.27 -> 32.67 | Improved |
| compare with Mamba2 on perplexity at 340M | Q-Delta · WikiText/LAMBADA, 340M | perplexity: Mamba2 26.60 / 30.35 vs Q-Delta 26.89 / 32.67 | Hurt |
| add query feedback on SQuAD recall task | Gated DeltaNet · Recall-intensive SQuAD, 340M | accuracy: 56.0 -> 52.3 | Hurt |
| add query feedback on SWDE and FDA recall tasks | Gated DeltaNet · Recall-intensive SWDE, FDA, 340M | accuracy: SWDE 29.1 -> 31.6; FDA 18.9 -> 22.0 | Improved |
| learn lambda_t end-to-end instead of fixed scalar 0.5 (ablation) | Q-Delta · Zero-shot commonsense avg, 340M | average accuracy: 47.20 -> 47.24 | Improved |
| use learnable lambda_t instead of fixed lambda = 0.8 (ablation) | Q-Delta · Zero-shot commonsense avg, 340M | average accuracy: 46.42 -> 47.24 | Improved |
| add adaptive gating of query feedback (vs lambda_t = 1) (ablation) | Q-Delta · LAMBADA ppl, 340M | perplexity: 35.21 -> 32.67 | Improved |
| add state decay gate alpha_t (ablation) | Q-Delta · Zero-shot commonsense avg, 340M | average accuracy: 45.86 -> 47.24 | Improved |
| fixed lambda on WikiText perplexity vs learnable (ablation) | Q-Delta · WikiText, 340M | perplexity: 26.89 vs 26.52-26.96 across variants | No clear effect |
Closest papers by meaning
- RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.86)
- Learning to Evict from Key-Value Cache (ICML 2026 · similarity 0.85)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.85)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.85)
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.84)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
