Research map / Architectures and efficiency / RNNs and normalization
A Framework for Inference Inspired by Human Memory Mechanisms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMI, a perception-memory-inference framework with dual working and long-term memory (LTM updated by outer-product association), instantiated as PMI-TR for Transformers and CNN_PMI for convolutional networks, evaluated on bAbI, Sort-of-CLEVR, equilateral triangle detection, language modeling and CIFAR-10.
Problem
Existing memory-augmented neural networks use monolithic or uniform memory, crude memory-memory relations, and shallow retrieval, limiting relational reasoning and long-term knowledge retention.
Approach
A dual-layer memory block where working memory is written by top-k sparse competitive cross-attention from perceptions, long-term memory is a 3D tensor updated by outer-product association with working memory, and inference retrieves from both memories and fuses them via multi-head cross-attention.
Key results
- PMI-TR reaches 2.55% mean test error on the 20 bAbI tasks (best run 2.32%), versus 3.6% for TR+HSW.
- PMI-TR achieves 0.96 test BPC on Enwik8 and 16.5 test perplexity on WikiText-103 (16-layer, 233M params) and 31.04 on PG-19.
- PMI-TR reaches 97.9% accuracy on equilateral triangle detection, an 8.1% improvement over the standard Transformer; CNN_PMI reaches 78.69% on CIFAR-10 versus 78.61% for CNN_MLP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the PMI dual-memory module to the Transformer | Transformer · bAbI-20k | mean error (%): 3.6±0.46 -> 2.55±0.11 | Helped |
| Add the PMI dual-memory module to the Transformer | Transformer · WikiText-103 test | perplexity: 17.1 -> 16.5 | Helped |
| Add the PMI dual-memory module to the Transformer | Transformer · Enwik8 test | bits-per-character: 0.97 -> 0.96 | Helped |
| Add the PMI dual-memory module to the Transformer | Transformer · PG-19 test | perplexity: 32.46 -> 31.04 | Helped |
| Add the PMI dual-memory module to ViT | ViT · CIFAR-10 | accuracy (%): 76.18 -> 79.12 | Helped |
| Add the PMI module to CNN_MLP | CNN_MLP · CIFAR-10 | accuracy (%): 78.61 -> 78.69 | No clear effect |
| Remove global memory sharing across layers (ablation) | PMI-TR_m · Sort-of-CLEVR binary and ternary | accuracy (%): 87.61 -> 80.29; 62.45 -> 60.06 | Hurt |
| Remove the LTM-to-WM correction step (ablation) | PMI-TR_m · Sort-of-CLEVR binary and ternary | accuracy (%): 87.61 -> 79.96; 62.45 -> 62.40 | Hurt |
| Remove WM involvement during inference (ablation) | PMI-TR_m · Sort-of-CLEVR ternary | accuracy (%): 62.45 -> 61.86 | Hurt |
| Replace top-k competition with standard soft competition (ablation) | PMI-TR_m · Sort-of-CLEVR binary and ternary | accuracy (%): 87.61 -> 79.64; 62.45 -> 61.87 | Hurt |
| Remove LTM guidance (ablation) | CNN_PMI · CIFAR-10 | accuracy (%): 78.69 -> 78.63 | No clear effect |
| Vary working-memory slots, LTM segments, and top-k setting (ablation) | PMI-TR_s, PMI-TR_m, and PMI-TR_l · bAbI and Sort-of-CLEVR | error (%) and accuracy (%) | Mixed |
Surprising findings
- Greater memory capacity does not necessarily yield better performance; the optimal setting was N=8 slots and M=5 LTM segments.
- Removing working memory from inference barely hurts bAbI (error rises only from 2.55 to 2.64), while removing LTM hurts more on Sort-of-CLEVR binary questions.
Limitations
- Gains reported on small-scale tasks and comparisons mostly against self-trained baselines; no large-scale pretrained LLM evaluation.
- Several claims about memory interpretation (attention visualizations, convergence of LTM) are qualitative and speculative.
Experimental setup
Datasets: bAbI-20k Sort-of-CLEVR Equilateral triangles (synthetic) Enwik8 WikiText-103 PG-19 CIFAR-10
Benchmarks: bAbI-20k Sort-of-CLEVR Equilateral triangle detection Enwik8 WikiText-103 PG-19 CIFAR-10
Metrics: error rate (%) accuracy (%) bits-per-character perplexity
Baselines: TR ISAB TR+HSW TR+HC STR LSTM DNC NUTM MemNet H-Mem Transformer-XL Compressive Transformer
Builds on: Transformer Vision Transformer Recurrent Memory Cell (RMC) Global Workspace Theory Shared workspace Transformer (TR+HSW)
Closest papers by meaning
- ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations (ICLR 2026 · similarity 0.88)
- REMem: Reasoning with Episodic Memory in Language Agent (ICLR 2026 · similarity 0.87)
- MemGen: Weaving Generative Latent Memory for Self-Evolving Agents (ICLR 2026 · similarity 0.87)
- Personal Visual Memory from Explicit and Implicit Evidence (NeurIPS 2026 · similarity 0.87)
- Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents (ICML 2026 · similarity 0.86)
- Dual-Latent Memory Routing for Vision-Language Reasoning (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.