Research map / Architectures and efficiency / Attention and transformers
GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
Yuri Kuratov, Matvey Kairov, Aydar Bulatov, Ivan Rodkin, Mikhail Burtsev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add second-order meta-learning through WRITE steps (ablation) | GradMem x1 · KV-retrieval (8 pairs) | exact match (%): 12.9 -> 99.7 | Improved |
| add second-order meta-learning through WRITE steps (ablation) | GradMem x2 · KV-retrieval (8 pairs) | exact match (%): 46.7 -> 100.0 | Improved |
| replace forward-only write with gradient-based write (K=1) | RMT, 8 memory vectors · KV-retrieval (16 pairs) | exact match (%): 45.5 -> 96.3 | Improved |
| increase gradient WRITE steps from 1 to 5 | GradMem · KV-retrieval (96 pairs) | exact match (%): 32.6 -> 88.4 | Improved |
| increase gradient WRITE steps from 1 to 2 | GradMem · KV-retrieval (64 pairs) | exact match (%): 58.6 -> 72.8 | Improved |
| repeat forward-only write twice | RMT · KV-retrieval (16 pairs) | exact match (%): 45.5 -> 69.6 | Improved |
| repeat forward-only write twice | RMT · KV-retrieval (32 pairs) | exact match (%): 44.3 -> 18.7 | Hurt |
| repeat forward-only write five times | RMT · KV-retrieval (32 pairs) | exact match (%): 44.3 -> 37.0 | Hurt |
| increase WRITE steps K | GradMem (GPT-2) · Short SQuAD | exact match (%): 38.1 -> 54.9 | Improved |
| increase WRITE steps K | GradMem (GPT-2) · WikiText-103 LM (tokens 128-255) | cross-entropy: 2.92 -> 2.91 | No clear effect |
| increase WRITE steps K | GradMem (GPT-2) · bAbI QA3 | exact match (%): 80.0 -> 79.3 | No clear effect |
| increase WRITE steps K | GradMem (GPT-2) · bAbI QA2 | exact match (%): 94.2 -> 93.9 | No clear effect |
Closest papers by meaning
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.88)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.87)
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.87)
- In-context Autoencoder for Context Compression in a Large Language Model (ICLR 2024 · similarity 0.87)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
