AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

IOT: Instance-wise Layer Reordering for Transformer Structures

Jinhua Zhu, Lijun Wu, Yingce Xia, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, Tie-Yan Liu

ICLR 2021 · Architectures and efficiency › Attention and transformers · approach: Transformer block and depth design · 3 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add instance-wise decoder layer orderingTransformer · IWSLT14 De→EnBLEU: 34.64 -> 35.62Improved
Use decoder order selection without encoder order selection (ablation)Transformer · IWSLT14 De→EnBLEU: 34.64 -> 35.60Improved
Use both encoder and decoder order selection (M=2, N=6) instead of decoder orders only (ablation)IOT · IWSLT14 De→EnBLEU: 35.60 -> 35.24Hurt
Vary decoder order count from N=2 to N=4 (ablation)IOT · IWSLT14 De→EnBLEU: 35.32 -> 35.62Improved
Add exploration and exploitation auxiliary losses with c1=0.1 and c2=0.01 (ablation)IOT (N=2) · IWSLT14 De→En devBLEU: 35.97 -> 36.60Improved
Replace auxiliary losses with temperature decay (ablation)IOT · not specifiedperformanceNo clear effect
Remove instance-wise selection and share reordered decoder parameters across the corpus (ablation)Transformer with shared decoder orders · IWSLT14 De→EnBLEU: 35.62 -> near 34.80Hurt
Add LayerDrop regularization (ablation)IOT (N=4) · IWSLT14 De→EnBLEU: 35.62 -> 36.13Improved
Apply instance-wise layer ordering to DynamicConvDynamicConv · IWSLT14 De→EnBLEU: 35.20 -> 35.87Improved
Train a Transformer with order 1, then decode using other layer orders (ablation)Transformer · IWSLT14 De→En devBLEU: 35.84 -> 0.35Hurt
Increase ordered decoders from N=2 to N=4 (ablation)IOT · WMT14 En→DeBLEU: 29.71 -> 30.03Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy