Research map / Architectures and efficiency / Attention and transformers
Incorporating BERT into Parallel Sequence Decoding with Adapters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BERT on both sides with encoder and cross-attention decoder adapters | AB-Net · IWSLT14 De-En | BLEU: 33.59 -> 36.49 | Helped |
| Use BERT on both sides with encoder and cross-attention decoder adapters | AB-Net · WMT14 De-En | BLEU: 32.69 -> 33.57 | Helped |
| Use parallel AB-Net decoding rather than Transformer-Base decoding | WMT14 En-De · WMT14 En-De | latency: 778* ms -> 327 ms | Helped |
| Add target-side BERT with decoder cross-attention adapters to the encoder-only variant (ablation) | AB-Net-Enc · IWSLT14 De-En | BLEU: 34.45 -> 36.49 | Helped |
| Add encoder adapters to Xbert + Decoder (ablation) | Xbert + Decoder · IWSLT14 De-En | BLEU | Helped |
| Add decoder adapters to Xbert + Aenc + Ybert (ablation) | Xbert + Aenc + Ybert · IWSLT14 De-En | BLEU | Helped |
| Add decoder adapters on all layers rather than only the top 6 layers (ablation) | AB-Net · IWSLT14 De-En | BLEU: 34.60 -> 36.49 | Helped |
| Add encoder adapters on all layers rather than only the top 6 encoder and decoder layers (ablation) | AB-Net · IWSLT14 De-En | BLEU: 33.78 -> 36.49 | Helped |
| Halve encoder-adapter hidden dimension from 2^9 to 2^8 (ablation) | AB-Net · IWSLT14 De-En | BLEU: 0.4 | Hurt |
| Use encoder-adapter hidden dimension d_Aenc=64 (ablation) | AB-Net · IWSLT14 De-En | BLEU: 34.81 | Helped |
| Fine-tune all AB-Net parameters instead of adapters only (ablation) | AB-Net · IWSLT14 De-En validation | convergence speed | Hurt |
| Train AB-Net from scratch instead of using pretrained BERT (ablation) | AB-Net · IWSLT14 De-En validation | convergence | Hurt |
Closest papers by meaning
- MASS: Masked Sequence to Sequence Pre-training for Language Generation (ICML 2019 · similarity 0.89)
- Incorporating BERT into Neural Machine Translation (ICLR 2020 · similarity 0.88)
- AMOM: Adaptive Masking over Masking for Conditional Masked Language Model (AAAI 2023 · similarity 0.88)
- Learning to Parallel: Accelerating Diffusion Large Language Models via Adaptive Parallel Decoding (ICLR 2026 · similarity 0.88)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.86)
- When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.