Research map / Architectures and efficiency / Attention and transformers
Memory Efficient Neural Processes via Constant Memory Attention Block
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMANPs, attentive neural processes built on a Constant Memory Attention Block whose cross-attention can be updated and computed in constant memory with respect to context size, plus an autoregressive not-diagonal variant.
Problem
Attention-based Neural Processes (TNPs, LBANPs, not-diagonal variants) require memory growing with context or target size, limiting use in low-resource settings and making context updates expensive.
Approach
Exact efficient update for cross-attention with a fixed learned query set (computed via log-space rolling log-sum-exp), enabling a stackable, permutation-invariant CMAB that computes outputs in constant memory and updates in constant computation per new point; an autoregressive block-wise not-diagonal extension (CMANP-AND) keeps memory constant in context size.
Key results
- CMANP-AND reaches 6.31 log-likelihood on CelebA 32x32 versus 5.57 for LBANP-ND, and it is the only not-diagonal model reported at CelebA 64x64 and 128x128 (6.96 and 7.15).
- On 1-D meta-regression (RBF), CMANP (vanilla) scores 1.24 vs LBANP 1.27, with constant memory versus linear memory for LBANP.
- CMHP matches THP on Reddit (RMSE 0.262 vs 0.238, ACC 0.609 vs 0.610) while using constant memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CMANP instead of LBANP | CMANP · CelebA 32x32 | log-likelihood: 3.97 ± 0.02 -> 3.93 ± 0.05 | No clear effect |
| Use CMANP instead of LBANP | CMANP · CelebA 64x64 and 128x128 | log-likelihood: 5.09 ± 0.02 -> 5.02 ± 0.14; 5.84 ± 0.01 -> 5.55 ± 0.01 | Hurt |
| Use CMANP-AND instead of LBANP-ND | CMANP-AND · CelebA 32x32 | log-likelihood: 5.57 ± 0.03 -> 6.31 ± 0.04 | Helped |
| Use CMANP-AND instead of TNP-ND | CMANP-AND · EMNIST Seen (0-9) | log-likelihood: 1.50 ± 0.00 -> 1.48 ± 0.03 | No clear effect |
| Use CMANP-AND instead of TNP-ND | CMANP-AND · EMNIST Unseen (10-46) | log-likelihood: 1.31 ± 0.00 -> 1.19 ± 0.03 | Hurt |
| Use CMANP instead of LBANP | CMANP · Meta-regression RBF and Matern 5/2 | log-likelihood: 1.27 ± 0.02 -> 1.24 ± 0.01; 0.85 ± 0.02 -> 0.80 ± 0.01 | Hurt |
| Use CMANP-AND instead of TNP-ND | CMANP-AND · Meta-regression RBF | log-likelihood: 1.46 ± 0.00 -> 1.48 ± 0.03 | Helped |
| Use CMANP-AND instead of TNP-ND | CMANP-AND · Meta-regression Matern 5/2 | log-likelihood: 1.02 ± 0.00 -> 0.96 ± 0.01 | Hurt |
| Replace Transformer layers with CMAB | THP on Reddit · Reddit | RMSE: 0.238 ± 0.028 -> 0.262 ± 0.037 | Hurt |
| Replace Transformer layers with CMAB | THP on Reddit · Reddit | NLL: 0.268 ± 0.098 -> 0.528 ± 0.209 | Hurt |
| Replace Transformer layers with CMAB | THP on Reddit · Reddit | ACC: 0.610 ± 0.002 -> 0.609 ± 0.003 | No clear effect |
| Reduce the prediction block size b_Q (ablation) | CMANP-AND · Image completion and meta-regression | log-likelihood | Helped |
Surprising findings
- The vanilla CMANP is slightly worse than LBANP on CelebA 64x64 and 128x128 only marginally differing; the paper notes performance drops on extreme contextual-bandit settings (delta 0.99, 0.995).
Limitations
- CMANP-AND is not target-equivariant, since its autoregressive predictions depend on the order of targets.
- Performance degrades at extreme contextual-bandit values of delta (0.99, 0.995), outside the training distribution.
- Results on EMNIST and meta-regression are competitive rather than uniformly best; TNP-D and TNP-ND lead on several benchmarks.
Experimental setup
Datasets: EMNIST CelebA GP-sampled regression tasks Reddit Mooc
Benchmarks: EMNIST CelebA 1-D meta-regression Reddit Mooc
Metrics: log-likelihood RMSE NLL ACC cumulative regret
Baselines: CNP NP ANP BANP TNP-D LBANP TNP-ND LBANP-ND
Builds on: Attentive Neural Processes Latent Bottlenecked Attentive Neural Processes Transformer Neural Processes Perceiver Transformer Hawkes Process
Closest papers by meaning
- Attentive Neural Processes (ICLR 2019 · similarity 0.85)
- Titans: Learning to Memorize at Test Time (NeurIPS 2025 · similarity 0.83)
- Rényi Neural Processes (ICML 2025 · similarity 0.83)
- Hyper Hawkes Processes: Interpretable Models of Marked Temporal Point Processes (NeurIPS 2026 · similarity 0.82)
- Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.82)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.