Research map / Architectures and efficiency / RNNs and normalization

Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives functional gradient learning rules for growing nonparametric networks whose layers are kernelized/weighted-softmax attention over a database of key-value pairs (RCDL), and shows MesaNet/DeltaNet-style dual views yield second-order-like optimizers.

Problem

Replacing linear attention in the dual form of gradient-trained linear layers with more powerful kernels (RBF, softmax) requires principled rules for which key-value pairs to append; naive softmax insertion breaks optimality.

Approach

Introduce a weighted softmax kernel whose Gaussian-mixture functional view admits smooth functional gradient descent that equals appending one weighted key-value tuple per step; for MesaNet/DeltaNet linear attention, the dual view yields an approximate second-order optimizer (FOOF-equivalent).

Key results

• On CIFAR-10 and CIFAR-100, growing weighted softmax and RBF models trained with plain SGD significantly outperform SGD-, Adam-, AdamW- and RMSProp-trained MLPs and are competitive with Muon-trained MLPs.
• In a single-pass CIFAR-10 regime, growing methods outperform all parametric models including Muon and Mesa.
• Replacing the feed-forward block in ViTs with an RCDL layer raises single-epoch CIFAR-10 accuracy from 48.4% to 54.4%.

Surprising findings

• Naively inserting the softmax into linear attention and applying gradient descent to values does not give a principled learning rule, because the new key alters the global softmax denominator.
• The parametric softmax baseline with matched width overfits with a widening train-test gap, whereas the growing weighted softmax does not.

Limitations

• Memory and compute grow linearly with the number of training datapoints.
• Advanced optimizers such as Adam and Muon are not yet derivable for the growing RCDL layers.
• Growing-model theory currently supports only vanilla SGD with weight decay and momentum.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 synthetic teacher-student regression data

Benchmarks: CIFAR-10 CIFAR-100 synthetic teacher-student regression

Metrics: test accuracy training loss single-pass accuracy memory growth

Baselines: SGD MLP Adam AdamW RMSProp Muon Mesa/FOOF optimizer k-nearest-neighbor SVM parametric softmax

Builds on: linear attention duality of gradient descent (fast weight programming) mesa-layer DeltaNet FOOF Moore-Aronszajn theorem

Closest papers by meaning

• Nonparametric Teaching of Attention Learners (ICLR 2026 · similarity 0.86)
• A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks (ICML 2024 · similarity 0.83)
• Neural Attention Search (NeurIPS 2025 · similarity 0.83)
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.83)
• NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.83)
• Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoders (ICLR 2024 · similarity 0.83)