Research map / Architectures and efficiency / RNNs and normalization
Hyperparameter Transfer for Dense Associative Memories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives and empirically validates muP-style hyperparameter transfer prescriptions (scales s1, s2 and learning rates) for dense associative memory networks with tied weights, using proportional scaling of N, K, P, B and centering of pre/post-activations.
Problem
Hyperparameter transfer across model and data scale is unexplored for Dense Associative Memories, whose tied weights across layers and steep activations break standard muP-type transfer rules.
Approach
Extends muP-style dimensional analysis to tied-weight DenseAMs, showing that proportional scaling of N, K, P, B makes training dynamics collapse, and that layer-wise centering of pre- and post-activations removes a mean-induced spike that otherwise breaks transfer for ReLU^p and softmax activations.
Key results
- Linear DenseAMs trained with SGD in the proportional regime show learning rate transfer and collapse of internal statistics when s1~1/sqrt(N), s2~1/sqrt(K), eta_W~K.
- Centering pre- and post-activations is necessary for ReLU DenseAM learning rate transfer; without it a rank-one spike grows with scale and destabilizes training.
- For softmax DenseAMs, SGD fails to give transfer while Adam with the proposed scaling does transfer across N, K, P, B.
Surprising findings
- Uncentered ReLU DenseAMs trained with SGD become unstable as width grows because of a mean-induced spike in the Gram matrix.
- SGD with softmax DenseAMs suffers numerical instability at large scale, whereas Adam does not.
Limitations
- Full analytic description of training dynamics is left to future work; derivations use isotropic data and first-order approximations.
- Empirical validation is on synthetic data and MNIST only; anisotropic transfer is empirical.
- Only shallow one-hidden-layer DenseAMs are studied.
Experimental setup
Datasets: MNIST synthetic Gaussian denoising data
Benchmarks: MNIST
Metrics: MSE loss learning rate transfer training dynamics collapse
Baselines: uncentered DenseAM
Builds on: muP Dense Associative Memory Krotov and Hopfield hidden-unit DenseAM Hopfield networks
Closest papers by meaning
- Dense Associative Memory Through the Lens of Random Features (NeurIPS 2024 · similarity 0.87)
- Dense Associative Memory with Epanechnikov Energy (NeurIPS 2025 · similarity 0.85)
- Semantically-correlated memories in a dense associative model (ICML 2024 · similarity 0.85)
- Memory Layers at Scale (ICML 2025 · similarity 0.84)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.84)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.