Research map / Architectures and efficiency / Attention and transformers
IOT: Instance-wise Layer Reordering for Transformer Structures
Jinhua Zhu, Lijun Wu, Yingce Xia, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, Tie-Yan Liu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add instance-wise decoder layer ordering | Transformer · IWSLT14 De→En | BLEU: 34.64 -> 35.62 | Improved |
| Use decoder order selection without encoder order selection (ablation) | Transformer · IWSLT14 De→En | BLEU: 34.64 -> 35.60 | Improved |
| Use both encoder and decoder order selection (M=2, N=6) instead of decoder orders only (ablation) | IOT · IWSLT14 De→En | BLEU: 35.60 -> 35.24 | Hurt |
| Vary decoder order count from N=2 to N=4 (ablation) | IOT · IWSLT14 De→En | BLEU: 35.32 -> 35.62 | Improved |
| Add exploration and exploitation auxiliary losses with c1=0.1 and c2=0.01 (ablation) | IOT (N=2) · IWSLT14 De→En dev | BLEU: 35.97 -> 36.60 | Improved |
| Replace auxiliary losses with temperature decay (ablation) | IOT · not specified | performance | No clear effect |
| Remove instance-wise selection and share reordered decoder parameters across the corpus (ablation) | Transformer with shared decoder orders · IWSLT14 De→En | BLEU: 35.62 -> near 34.80 | Hurt |
| Add LayerDrop regularization (ablation) | IOT (N=4) · IWSLT14 De→En | BLEU: 35.62 -> 36.13 | Improved |
| Apply instance-wise layer ordering to DynamicConv | DynamicConv · IWSLT14 De→En | BLEU: 35.20 -> 35.87 | Improved |
| Train a Transformer with order 1, then decode using other layer orders (ablation) | Transformer · IWSLT14 De→En dev | BLEU: 35.84 -> 0.35 | Hurt |
| Increase ordered decoders from N=2 to N=4 (ablation) | IOT · WMT14 En→De | BLEU: 29.71 -> 30.03 | Improved |
Closest papers by meaning
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.86)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.86)
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.85)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.85)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.85)
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
