Research map / Continual learning / Online and replay-based
Predictive Attractor Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Predictive Attractor Models (PAM), a biologically plausible, Hebbian-trained, streaming sequence memory that avoids catastrophic forgetting and samples among multiple future continuations via attractors.
Problem
Sequential memory in biologically plausible networks suffers from catastrophic forgetting, limited capacity, slow iterative learning, low-order Markov context, and inability to represent and sample multiple valid future continuations from the same context.
Approach
Combines an HTM-inspired predictor with lateral inhibition in minicolumns (one active context neuron per minicolumn) and a Hopfield-style attractor model derived from a GMM state-space formulation; learning uses local Hebbian rules trained online, one pass per transition, and generation samples from the union of predicted SDRs.
Key results
- PAM retains previously learned sequences under continual learning (BWT) where tPC fails regardless of correlation, with PAM needing larger context neurons N_k for highly correlated ProteinNet data.
- In offline sequence capacity, PAM outperforms tPC, AHN and HN and its capacity increases with context neurons N_k while staying robust to sequence correlation.
- PAM runs on CPU and is faster than single-layer tPC, while two-layer tPC takes two to three orders of magnitude more time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase context neurons per minicolumn (Nk) (ablation) | PAM · Offline sequence capacity | maximum sequence length | Helped |
| increase sequence correlation (ablation) | PAM · Offline sequence capacity | maximum sequence length | No clear effect |
| increase polynomial degree of the separation function (ablation) | Hopfield Network · Offline sequence capacity | maximum sequence length | Helped |
| increase sequence correlation (ablation) | Hopfield Network · Offline sequence capacity | maximum sequence length | Hurt |
| use high enough context capacity rather than Nk=1 (ablation) | PAM · Synthetic sequences; ProteinNet | backward transfer | Helped |
| learn sequences continually, one word at a time, rather than in a batched setup | PAM · Four-letter English words | average IoU | Helped |
| use attractors to clean noisy observations with prior beliefs | PAM · Online sequence generation; CLEVRER | sequence recall | Helped |
| increase input noise (ablation) | PAM · CLEVRER | sequence recall | Hurt |
| use two layers instead of one in temporal predictive coding (ablation) | Temporal Predictive Coding · CIFAR sequences | context prediction | Mixed |
| increase the tPC model from one layer to two layers (ablation) | Temporal Predictive Coding · Sequence learning and recall | runtime | Hurt |
| use a single-layer tPC model instead of PAM | Sequence learning and recall · CPU runtime | runtime | Hurt |
Surprising findings
- Two-layer tPC attempts to average its predictions in highly correlated sequences, and single-layer tPC alternates indefinitely between patterns.
- PAM is robust to input noise: it replaces noisy inputs with its prior belief when they are not in the predicted set, while other methods use the noisy inputs.
Limitations
- Requires inputs in sparse binary SDR format; cannot directly process dense images, relying on an SDR autoencoder for vision.
- Evaluated on small synthetic and simplified datasets (e.g., 100 four-letter words) rather than large-scale real tasks.
- Baselines were trained in batched setups while PAM learns online, a disadvantage the authors note, and hyperparameters were optimized once per method.
Experimental setup
Datasets: ProteinNet Moving MNIST CLEVRER CIFAR synthetic SDR sequences most frequent English 4-letter words
Benchmarks: Moving MNIST CLEVRER CIFAR ProteinNet four-letter English words
Metrics: normalized IoU Backward Transfer (BWT) MSE sequence capacity running time
Baselines: Temporal Predictive Coding (tPC) 2-layer tPC Asymmetric Hopfield Network (AHN) Hopfield Network (HN)
Builds on: Hierarchical Temporal Memory Temporal Predictive Coding Asymmetric Hopfield Network Universal Hopfield Networks Variational Free Energy
Closest papers by meaning
- Sequential Memory with Temporal Predictive Coding (NeurIPS 2023 · similarity 0.91)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.89)
- Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks (NeurIPS 2025 · similarity 0.87)
- Rapid Learning without Catastrophic Forgetting in the Morris Water Maze (ICML 2024 · similarity 0.86)
- Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture (ICML 2024 · similarity 0.86)
- PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.