AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Reversible Recurrent Neural Networks

Matthew MacKay, Paul Vicol, Jimmy Ba, Roger B Grosse

NeurIPS 2018 · Architectures and efficiency › RNNs and normalization · approach: Efficient recurrent architectures · 20 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use reversible recurrent transitions with information bufferingGRU language models on Penn TreeBank · Penn TreeBank validationperplexity: 82.2 -> 81.5No clear effect
Use reversible recurrent transitions with information bufferingLSTM language models on WikiText-2 · WikiText-2 validationperplexity: 89.3 -> 94.1Hurt
Use Emb+20H attention and reversible recurrenceGRU translation model · Multi30K testBLEU: 32.60 -> 34.41Improved
Use Emb+20H attention and reversible recurrenceLSTM translation model · Multi30K testBLEU: 37.06 -> 37.23Improved
Use reversible recurrenceGRU translation model · IWSLT 2016 testBLEU: 16.07 -> 20.70Improved
Use reversible recurrenceLSTM translation model · IWSLT 2016 testBLEU: 22.35 -> 22.34No clear effect
Restrict forgetting to 2–3 bits per hidden unit per timestep (ablation)RevGRU and RevLSTM language models · Penn TreeBank validationperplexityNo clear effect
Restrict forgetting to at most 1 bit per hidden unit per timestep (ablation)RevGRU language model · Penn TreeBank validationperplexity: 4–5 increase in perplexityHurt
Use discrete instead of fractional forgetting (ablation)Reversible language models · Penn TreeBank validationperplexity: 4-10 perplexity pointsHurt
Use straight-through Gumbel-Softmax instead of straight-through categorical gradient estimation (ablation)Discrete-forgetting models · Penn TreeBank validationperplexity: 2--3 perplexityImproved
Use attention over embeddings and a 20-dimensional hidden-state slice instead of other attention variants (ablation)Reversible NMT models · Multi30K testBLEUMixed
Remove the forgetting step (ablation)Reversible recurrent models · Penn TreeBank language modelingperplexityHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy