Research map / Architectures and efficiency / Attention and transformers

Cached Transformers: Improving Transformers with Differentiable Memory Cachde

AAAI 2024 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Cached Transformer with Gated Recurrent Cache (GRC) attention, a differentiable, gated memory of compressed past token embeddings that is attended alongside current tokens, evaluated on language, translation, LRA, ImageNet classification and COCO detection.

Problem

Standard self-attention is quadratic in sequence length and cannot cheaply capture long-range dependencies; existing memory/cache methods are FIFO-limited or require costly KNN search over large token caches.

Approach

A fixed-size Gated Recurrent Cache updated via reset and update gates compresses the history of token embeddings into vectors; attention output is an learned interpolation of cached attention and self-attention, so the cache cost is independent of the number of cached tokens.

Key results

• On ImageNet-1k, GRC-cached PVT-Tiny reaches 78.4% top-1 vs 75.1% baseline (+3.3), and cached models gain 0.5-3.3 top-1 across ViT, PVT, Swin and PVTv2.
• On WikiText-103, GRC-cached Transformer-XL base reaches 22.9 test PPL vs 24.0 baseline, and on Long ListOps GRC adds 1.17 (Transformer), 1.39 (BigBird) and 0.58 (Reformer) accuracy points.
• On COCO with Mask R-CNN 1x, cached PVT-Medium improves box AP from 42.0 to 46.6 (+4.6).

Measured effects reported in the paper

Surprising findings

• Attention-based (Transformer-XL-style) caches barely improve or even hurt ImageNet accuracy (e.g., PVT-Tiny 74.8 vs 75.1), while GRC gives large gains.
• Learned gates favor cached attention in more than half of layers, with cached attention dominating in early layers and self-attention in the last layers.

Limitations

• Adds roughly 10-20% FLOPs and throughput cost over baselines.
• Cache ratio and memory length are chosen by preliminary experiments; gains saturate for ratios above 0.5.
• Cache is averaged across the batch and frozen at inference, so it is a dataset-level running state rather than per-sample memory.

Experimental setup

Datasets: ImageNet-1k ImageNet-22k COCO train2017 COCO val2017 WikiText-103 IWSLT14 IWSLT15 Long ListOps (LRA)

Benchmarks: ImageNet-1k ImageNet-22k COCO val2017 WikiText-103 IWSLT14 IWSLT15 Long ListOps (LRA)

Metrics: top-1 accuracy top-5 accuracy box AP mask AP test perplexity BLEU accuracy

Baselines: Transformer-XL Memory Transformer (MT) Recurrent Memory Transformer (RMT) ViT PVT PVTv2 Swin Reformer BigBird Mask R-CNN

Builds on: Transformer-XL Reformer BigBird PVT Mask R-CNN

Closest papers by meaning

• Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.86)
• Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.86)
• An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.86)
• Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.86)
• LASER: Attention with Exponential Transformation (ICML 2025 · similarity 0.86)
• Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.85)