Research map / Architectures and efficiency / Attention and transformers
Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hybrid quadratic-linear transformers (HQLTs) that blend softmax-attention KV-memory with DeltaNet fast-weight memory in a single layer, comparing delayed-streaming, delayed-chunk, and synchronous blending strategies.
Problem
Quadratic transformers offer precise retrieval but are limited to bounded context, while linear transformers/fast weight programmers handle long sequences and state tracking but lose precise recall; the paper asks how to combine these complementary memory systems within one layer.
Approach
Within a single layer, combine sliding-window softmax KV-memory with DeltaNet FW-memory, and find that synchronous feeding of the same key-value pair to both memories (with dynamic vector mixing) preserves DeltaNet's state-tracking expressivity while retaining KV retrieval, unlike delayed variants that feed old KV pairs to FW-memory.
Key results
- Synchronous HQLT at 1.3B reaches 53.9 average on six common-sense tasks vs 52.3 for DeltaNet and 53.0 for Transformer++, with LAMBADA perplexity 15.9 vs 19.9 (DeltaNet) and 17.9 (Transformer++).
- Synchronous HQLT solves parity (100.0 normalized accuracy) and modular arithmetic (97.0) where delayed variants and the Transformer fail.
- On retrieval tasks, a 340M Synchronous HQLT with window 1024 reaches 40.4 average vs 29.0 for Transformer++ with the same window, but still trails the full-context Transformer++ (44.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use synchronous rather than delayed memory updates | HQLT · Parity | normalized accuracy: 3.3 -> 100.0 | Helped |
| use synchronous rather than delayed memory updates | HQLT · Modular Arithmetic | normalized accuracy: 27.8 -> 97.0 | Helped |
| replace DeltaNet with vanilla linear attention (ablation) | HQLT Synchronous · Parity | normalized accuracy: 100.0 -> 2.5 | Hurt |
| replace DeltaNet with vanilla linear attention (ablation) | HQLT Synchronous · General language modeling, 340M | accuracy average: 47.8 -> 42.2 | Hurt |
| increase KV-memory window from 64 to 128 (ablation) | HQLT Synchronous, 340M · SWDE | accuracy: 20.0 -> 16.9 | Hurt |
| increase KV-memory window from 64 to 512 (ablation) | HQLT Synchronous, 340M · SWDE | accuracy: 20.0 -> 22.9 | Helped |
| increase KV-memory window from 64 to 128 (ablation) | HQLT Synchronous, 340M · FDA | accuracy: 10.9 -> 17.9 | Helped |
| increase KV-memory window from 128 to 256 (ablation) | HQLT Synchronous, 340M · FDA | accuracy: 17.9 -> 15.1 | Hurt |
| increase KV-memory window from 64 to 1024 (ablation) | HQLT Synchronous, 340M · retrieval average | accuracy average: 19.8 -> 40.4 | Helped |
| replace dynamic vector mixing with sum mixing (ablation) | HQLT Synchronous, 340M · LAMBADA | perplexity: 29.4 -> 34.8 | Hurt |
| remove RoPE positional encoding from KV-memory (ablation) | HQLT Synchronous · general language modeling | perplexity: 26.3 -> 27.3 | Hurt |
| remove RoPE positional encoding from KV-memory (ablation) | HQLT Synchronous, 340M, window 1024 · FDA | accuracy: 50.1 -> 55.8 | Helped |
Surprising findings
- Delayed-Stream and Delayed-Chunk variants fail on parity and modular arithmetic, just like the quadratic transformer, showing that feeding FW-memory with delayed inputs removes its expressivity benefit.
- Increasing the KV window from 64 to 512 has minimal effect on general language modeling performance.
Limitations
- Retrieval performance remains weak; a large KV window appears necessary for precise retrieval, and HQLT still trails the full-context Transformer on retrieval tasks.
- Only the basic DeltaNet update rule is studied; other update rules (gated delta, delta product) are not explored.
- Retrieval results are sensitive to evaluation protocol and some window-size trends are mixed.
Experimental setup
Datasets: FineWeb-Edu WikiText-2 LAMBADA PIQA HellaSwag WinoGrande ARC-easy ARC-challenge SWDE SQuAD FDA parity
Benchmarks: WikiText-2 LAMBADA PIQA HellaSwag WinoGrande ARC-easy ARC-challenge SWDE SQuAD FDA parity modular arithmetic
Metrics: perplexity accuracy normalized accuracy average return success rate
Baselines: Transformer++ DeltaNet Linear Attention Mamba Transformer Infini-attention
Builds on: DeltaNet Infini-attention flash-attention flash-linear-attention Transformer++
Closest papers by meaning
- B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory (NeurIPS 2024 · similarity 0.86)
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length (NeurIPS 2024 · similarity 0.86)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.85)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.85)
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.85)
- xLSTM: Extended Long Short-Term Memory (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.