AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction

Muralidhar Andoorveedu, Zhanda Zhu, Bojian Zheng, Gennady Pekhimenko

NeurIPS 2022 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply Tempo (In-place GELU, In-place LayerNorm, dropout recomputation) vs BaselineBERT-Large · 2080 Ti seq len 512throughput speedup: 16%Improved
apply Tempo vs Baseline on larger max batch sizeBERT-Large · 2080 Ti seq len 512max batch size: 1 -> 2Improved
apply Tempo vs Baseline on V100BERT-Large · V100 seq len 512max batch size: 4 -> 7Improved
use default layer checkpointing instead of BaselineBERT-Large · 2080 Ti seq len 128throughput: 142.3160342 -> 120.0995009Hurt
Tempo versus Checkpoint on memory footprint at batch size 15BERT-Large · 2080 Ti seq len 128total memory: 8.3 GB -> 9.2 GBHurt
Tempo versus Baseline on memory at batch size 15BERT-Large · 2080 Ti seq len 128total memory: 11.3 GB -> 9.2 GBImproved
Tempo versus Baseline on pre-training lossBERT-Base · pre-training phase 1loss: 0.5%No clear effect
Tempo versus Baseline on fine-tuning accuracyBERT-Large · MRPCaccuracyNo clear effect
run on A100 with larger hidden sizes (ablation)Tempo · A100 BERT H=2048/3072throughput speedup: 39%Improved
run on longer sequence lengths up to 3072 on A100 (ablation)Tempo · BERT-Large 12 layersthroughput speedup: 27%Improved
apply Tempo to GPT2 and RoBERTa (ablation)Tempo · 2080 Tithroughput speedup: 19% and 26%Improved
apply Tempo to GPT2 and RoBERTa on V100 (ablation)Tempo · V100throughput speedup: 5% and 4%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy