Research map / Architectures and efficiency / RNNs and normalization

Nested Learning: The Illusion of Deep Learning Architectures

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Neural-symbolic reasoning · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Nested Learning, a paradigm that represents models and their optimizers as nested multi-level associative-memory optimization problems, and uses it to reinterpret optimizers (Adam, momentum) and design new ones (Delta GD, M3) and a self-modifying continual-learning module (HOPE).

Problem

Deep learning models are largely static after deployment and cannot continually learn; existing architectures and optimizers are treated separately, and the notion of depth and in-context learning is not unified.

Approach

Model architectures and their optimizers are jointly represented as nested systems of associative memories, each with its own gradient flow and update frequency; optimizers like Adam and momentum are reinterpreted as memories compressing gradients, enabling more expressive optimizers and a continuum memory system with multi-frequency updates.

Key results

• The excerpt reports no concrete numerical results; experimental evaluations on language modeling, continual learning, and long-context tasks are only listed in the roadmap.

Surprising findings

• Transformers are reinterpreted as two-frequency systems (update rates infinity and zero for attention and MLP), and Adam is argued to be an optimal associative memory under L2 regression.

Limitations

• Experimental results are not included in the provided excerpt, so claims about the proposed modules are unverified here.
• Many optimizer reinterpretations rely on approximations (e.g., element-wise and simplified preconditioners).

Experimental setup

Benchmarks: needle-in-a-haystack BABILong

Metrics: accuracy

Baselines: Transformer Adam AdaGrad SGD with momentum

Builds on: Fast weight programmers Linear Transformers MAML Adam Titans

Closest papers by meaning

• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.87)
• Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.87)
• In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.86)
• Probing the Decision Boundaries of In-context Learning in Large Language Models (NeurIPS 2024 · similarity 0.86)
• The mechanistic basis of data dependence and abrupt learning in an in-context classification task (ICLR 2024 · similarity 0.85)
• Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models (ICLR 2025 · similarity 0.85)