Research map / Bandits and online learning / Online convex optimization and games
Learning Successor Features with Distributed Hebbian Temporal Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distributed Hebbian Temporal Memory (DHTM), a factor-graph, multi-compartment neuron model trained with local Hebbian-like rules that forms successor features for online RL in partially observable, non-stationary Gridworld and AnimalAI environments.
Problem
Online sequence learning for decision making under uncertainty in non-stationary, partially observable environments, where RNNs and LSTMs rely on unstable backpropagation and require full sequences.
Approach
A factor-graph formalism combined with HTM-style dendritic segments, where transition factors are stored as segments and updated by one-step Monte Carlo Hebbian rules, yielding an episodic-like memory that predicts cumulative feature distributions (successor features) fully online.
Key results
- On Gridworld, DHTM and table episodic control adapt online to the blocked path, while CSCG, LSTM and RWKV fail to adapt as well.
- On AnimalAI, DHTM reaches the goal stably after about 150 episodes (3.5e3 action steps), while a small Dreamer V3 (18M) needs about 1000 episodes (35e3 action steps).
- LSTM and RWKV show no learning on the AnimalAI time-step scale; CSCG improves only slowly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DHTM instead of Dreamer V3 (18M) | AnimalAI agent · AnimalAI | action steps to solve: 35e3 versus 3.5e3 | Helped |
| use context factor size greater than 1 instead of factor size 1 (ablation) | DHTM · AnimalAI, Categorical VAE | steps to reach goal | Helped |
| use factor 1* with three hidden states per feature variable instead of factor size 1 (ablation) | DHTM · AnimalAI, Categorical VAE | steps to reach goal | Hurt |
| use context factor size 1 (ablation) | DHTM · AnimalAI, Categorical VAE | steps to reach goal | Hurt |
Surprising findings
- Table episodic control performs on par with DHTM on the Gridworld task, suggesting its mechanism explains most of DHTM's performance there.
- A context factor size of 1 (equivalent to parallel HMMs) fails to converge to the shortest path on the distributed VAE encoding in AnimalAI.
Limitations
- Evaluation limited to basic RL setups with a simple agent architecture and simple exploration.
- Segment growth may become excessive in noisy environments; policy becomes less stable as Gridworld size grows.
- Hidden state uniquely encodes each trajectory, so no generalization across different trajectories reaching the same state.
Experimental setup
Datasets: Gridworld (homebrew 5x5 POMDP maze) AnimalAI 10x10 room RGB observations
Benchmarks: Gridworld 5x5 alternating maze AnimalAI 10x10 room
Metrics: action steps to reach goal total steps over 1000 episodes sample efficiency
Baselines: CSCG LSTM RWKV Episodic Control (table) Dreamer V3
Builds on: Successor Features Hierarchical Temporal Memory Factorial HMM Baum-Welch CSCG
Closest papers by meaning
- Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (ICLR 2024 · similarity 0.85)
- Predictive Attractor Models (NeurIPS 2024 · similarity 0.83)
- Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning (NeurIPS 2025 · similarity 0.83)
- What Do Hebbian Learners Learn? Reduction Axioms for Iterated Hebbian Learning (AAAI 2024 · similarity 0.82)
- Multi-View Incremental Learning with Structured Hebbian Plasticity for Enhanced Fusion Efficiency (AAAI 2025 · similarity 0.82)
- Spike-timing-dependent Hebbian learning as noisy gradient descent (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.