Research map / Architectures and efficiency / Pruning and efficient tuning
Mnemosyne: Learning to Train Transformers with Transformers
Deepali Jain, Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Sumeet Singh, Vikas Sindhwani, Tingnan Zhang, Jie Tan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LSTM memory with Mnemosyne's spatio-temporal attention | learned optimization of neural networks · ViT training on MNIST, CIFAR10, and CIFAR100 | training loss | Improved |
| use Mnemosyne instead of the LSTM-based VeLO optimizer | learned optimization of neural networks · ViT training | training performance | Improved |
| use coordinate-wise Mnemosyne instead of tuned Adam variants | ViT-H(32) last-layer fine-tuning · ImageNet2012, Places365, Caltech-Birds-2011 | accuracy | Mixed |
| use light Mnemosyne for part of the model while retaining Adam for other layers | ViT-B(16) fine-tuning · CIFAR100 | accuracy | Improved |
| use coordinate-wise Mnemosyne instead of Adam variants | T5XXL soft prompt-tuning · SuperGLUE | performance | Improved |
| use tensor-wise Mnemosyne instead of Adam variants | BERT-base MLM pre-training · masked language modeling | performance | Mixed |
| use tensor-wise Mnemosyne instead of optimal Adam | ViT-H(32) fine-tuning · CIFAR100, Places365, ImageNet | performance | Mixed |
| combine coordinate-wise and tensor-wise Mnemosyne | Super-Mnemosyne · ViT fine-tuning | performance | Improved |
| replace compact associative memory with regular Transformer attention (ablation) | Mnemosyne temporal module · optimizer meta-training | performance | Hurt |
| use FAVOR++ random features instead of FAVOR+ (ablation) | Mnemosyne CAM · optimizer meta-training | convergence speed | Improved |
| use FAVOR++ random features instead of FAVOR+ (ablation) | Mnemosyne CAM · optimizer meta-training | asymptotic performance | No clear effect |
| use hyperbolic cosine random features instead of regular positive random features (ablation) | Mnemosyne CAM · optimizer learning | variance | Improved |
Closest papers by meaning
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.86)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.86)
- Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner (NeurIPS 2024 · similarity 0.86)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
