Research map / Architectures and efficiency / Attention and transformers
TNT: Improving Chunkwise Training for Test-Time Memorization
Zeman Li, Ali Behrouz, Yuan Deng, Peilin Zhong, Praneeth Kacham, Mahdi Karami, Meisam Razaviyayn, Vahab Mirrokni
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add global memory to hierarchical TNT (ablation) | TNT Stage 1 · language modeling | perplexity: 25.60 -> 21.04 | Improved |
| add Q-K projection to local retrieval (ablation) | TNT Stage 1 · language modeling | perplexity: 22.01 -> 21.04 | Improved |
| add local memory modules (1 to 4) (ablation) | TNT Stage 1 · language modeling | perplexity: 23.53 -> 20.15 | Improved |
| add Stage 2 fine-tuning with smaller local chunk (ablation) | TNT Stage 1 · language modeling | perplexity: 21.04 -> 20.86 | Improved |
| add more local memories (ablation) | TNT Stage 1 · commonsense reasoning | average accuracy: 41.0 (N=2) vs 40.2 (N=3) vs 40.6 (N=4) | Mixed |
| use TNT hierarchical memory instead of plain Titans | Titans · 150M avg perplexity | perplexity: 25.07 -> 23.13 | Improved |
| use TNT instead of Titans at same chunk size 8 | Titans · time to loss 3.20 | training time (hrs): 19.48 -> 2.54 | Improved |
| TNT versus gated Transformer on perplexity | Gated Transformer · 150M avg perplexity | perplexity: 22.39 vs 23.09 (TNT worse) | Hurt |
| increase Titans chunk size from 8 to 128 (ablation) | Titans · time to loss 3.20 | training time (hrs): 19.48 -> 3.71 | Improved |
Closest papers by meaning
- Rethinking Expressivity and Efficiency in Test-Time Training (NeurIPS 2026 · similarity 0.88)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.87)
- Learning What to Remember: Test-Time Training via Context Distillation (NeurIPS 2026 · similarity 0.87)
- Titans: Learning to Memorize at Test Time (NeurIPS 2025 · similarity 0.86)
- In-Place Test-Time Training (ICLR 2026 · similarity 0.85)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
