Research map / Reinforcement learning / Visual and representation RL

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

ICLR 2025 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Stable Hadamard Memory (SHM), a Hadamard-product memory writing model with input-conditioned calibration matrices built from randomly sampled trainable parameters, and reports gains over GRU, FFM and other memory models on meta-RL, credit assignment and POPGym.

Problem

Memory writing in memory-augmented agents for partially observable, long-horizon RL is unstable (gradient explosion/vanishing) and inefficient, and existing matrix memories lack flexible forgetting and recall.

Approach

Unifies memory writing as M_t = M_{t-1} ⊙ C_t + U_t (Hadamard Memory Framework) and sets C_t = 1 + tanh(θ_t ⊗ v_c(x_t)) with θ_t randomly sampled from a learnable set, so calibration is input-conditioned and its cumulative product has expectation near 1, reducing vanishing/exploding behavior.

Key results

• On POPGym hardest games, SHM averages -5.1 return versus -24.2 for FFM and -28.4 for GRU (about 10-12% relative improvement over FFM and GRU as reported).
• On Visual Match, SHM perfectly solves both 250- and 500-step versions, while the second-best FFM reaches 77% and 25% success.
• In the easy meta-RL mode, SHM achieves near-optimal success while other methods underperform by 20-50% on average.

Measured effects reported in the paper

Surprising findings

• Fixed calibration matrices cause the cumulative product to vanish within about 10 steps, and random-C calibration without learned structure underperforms.
• SHM is slightly slower per batch than GRU and FFM (1.9 ms vs 1.6 and 1.8 ms) despite the claimed efficiency of the parallel formulation, since the non-parallel implementation was used.

Limitations

• Stability guarantee is not perfect; the expectation argument relies on independence assumptions that RL data violates.
• Meta-RL and credit assignment experiments used small memory sizes (512 elements) due to slow codebase, limiting SHM's potential.
• Non-parallel implementation makes SHM slightly slower than GRU and FFM.

Experimental setup

Datasets: Wind Point Robot Visual Match Key-to-Door POPGym Autoencode POPGym Battleship POPGym Concentration POPGym RepeatPrevious

Benchmarks: Wind Point Robot Visual Match Key-to-Door POPGym

Metrics: success rate episode return

Baselines: GRU FWP GPT-2 S6 mLSTM FFM

Builds on: Neural Turing Machine Differentiable Neural Computer Fast Weight Programmers Fast and Forgetful Memory POPGym

Closest papers by meaning

• Mem-T: Densifying Rewards for Long-Horizon Memory Agents (ICML 2026 · similarity 0.90)
• Hadamard Representation: Scaffolding Performance Across Model-free RL (NeurIPS 2026 · similarity 0.89)
• MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory (NeurIPS 2026 · similarity 0.88)
• Mastering Memory Tasks with World Models (ICLR 2024 · similarity 0.88)
• MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (ICLR 2026 · similarity 0.88)
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning (NeurIPS 2025 · similarity 0.88)