AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Mnemosyne: Learning to Train Transformers with Transformers

Deepali Jain, Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Sumeet Singh, Vikas Sindhwani, Tingnan Zhang, Jie Tan

NeurIPS 2023 · Architectures and efficiency › Pruning and efficient tuning · approach: Efficient ViT adaptation · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace LSTM memory with Mnemosyne's spatio-temporal attentionlearned optimization of neural networks · ViT training on MNIST, CIFAR10, and CIFAR100training lossImproved
use Mnemosyne instead of the LSTM-based VeLO optimizerlearned optimization of neural networks · ViT trainingtraining performanceImproved
use coordinate-wise Mnemosyne instead of tuned Adam variantsViT-H(32) last-layer fine-tuning · ImageNet2012, Places365, Caltech-Birds-2011accuracyMixed
use light Mnemosyne for part of the model while retaining Adam for other layersViT-B(16) fine-tuning · CIFAR100accuracyImproved
use coordinate-wise Mnemosyne instead of Adam variantsT5XXL soft prompt-tuning · SuperGLUEperformanceImproved
use tensor-wise Mnemosyne instead of Adam variantsBERT-base MLM pre-training · masked language modelingperformanceMixed
use tensor-wise Mnemosyne instead of optimal AdamViT-H(32) fine-tuning · CIFAR100, Places365, ImageNetperformanceMixed
combine coordinate-wise and tensor-wise MnemosyneSuper-Mnemosyne · ViT fine-tuningperformanceImproved
replace compact associative memory with regular Transformer attention (ablation)Mnemosyne temporal module · optimizer meta-trainingperformanceHurt
use FAVOR++ random features instead of FAVOR+ (ablation)Mnemosyne CAM · optimizer meta-trainingconvergence speedImproved
use FAVOR++ random features instead of FAVOR+ (ablation)Mnemosyne CAM · optimizer meta-trainingasymptotic performanceNo clear effect
use hyperbolic cosine random features instead of regular positive random features (ablation)Mnemosyne CAM · optimizer learningvarianceImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy